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Preface 


For the fourtieth time, the Association for Educational Communications and Technology (AECT) is 
sponsoring the publication of these Proceedings. Papers published in this volume were presented at the 
annual AECT Convention in Kansas City, MO. A limited quantity of these Proceedings were printed and 
sold in both hardcopy and electronic versions. Volumes | and 2 are available through the Educational 
Resources Clearinghouse (ERIC) System. Proceedings volumes are available to members at AECT.ORG. 
Proceedings copies are also available at: 


http://www.tresystems.com/proceedings/ 


The Proceedings of AECT’s Convention are published in two volumes. Volume #1 contains papers dealing 
primarily with research and development topics. Papers dealing with the practice of instructional 
technology including instruction and training issues are contained in Volume #2. This year, both volumes 
are included in one document. 


REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and included in these 
Proceedings were subjected to a reviewing process. All references to authorship were removed from 
proposals before they were submitted to referees for review. Approximately sixty percent of the 
manuscripts submitted for consideration were selected for presentation at the convention and for 
publication in these Proceedings. The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 


Michael R. Simonson 
Deborah J. Seepersaud 
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Using Technology to Improve Migrant Adult Education Instruction 


Melissa Adams 
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Descriptors: Migrant Adult Education (Latinx), Technology 


Abstract: 


The GED (General Educational Development or General Equivalency Diploma) changed in 2014 to be 
computer based. Since then there has been a growing need for more educational technology in the GED/HEP 
classrooms. While several computer-based programs for the English GED exist, there have been few available for 
the Spanish GED. Since majority of HEP participants are Spanish Speakers, there is a greater need for Spanish 
computer-based programs in the HEP classroom. This presentation will focus on the need of Spanish computer- 
based programs in the HEP classroom to prepare the students for the GED exam. 


Description 


The roundtable session focuses on the needs of migrant adult educational programs. These programs are 
the HEP (High School Equivalency) programs which are funded through the Department of Education: Office of 
Migrant Education to educate migrant and seasonal farm workers. Since most of this population needs bilingual 
(English and Spanish) instruction, there is a need to incorporate more Educational Technology bilingually into the 
HEP classroom. The session will include a discussion of the bilingual tools and technologies available. 


Disclaimer 


For the purpose of this study the Hispanic and Latino (a) population will be referred to as Latinx. 
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Abbreviations Frequently Used in this Study 
HEP: High School Equivalency Program 
GED: General Education Development/Diploma 
OME: Office of Migrant Education 
History of GED and Latinx 
The GED test was officially created in 1942 which measured English language arts, social studies, 


mathematics, and science. There were three more versions of the GED exam, 1978, 1988, and 2002 series which 
kept the original content from the 1942 exam with increased levels of proficiency of the assessments (GED Testing 
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Service, 2018). These exams were available in English, Spanish and French. In 2014, the GED changed drastically 
and created many challenges for adult education. The GED2014 exam was completely redesigned from paper-based 
test to computer-based test which included a new scoring system and a higher level of content in English language 
arts, social studies, mathematics, and science (GED Testing Service, 2018). The GED2014 is only offered in 
English and Spanish currently. 


In the late 1950s and early 1960s, it was a difficult time in education for the Latinx population because 
advancement was challenging for them (Arciniega, 2012). In the late 1960s, 1970s, and 1980s, changes were made 
in the acceptance of the education of the Latinx population which resulted in more Latinx people as students, 
teachers, and department heads and included more acceptance than the previous decade (Arciniega, 2012). Since 
1971, GED attainment continually increased due to minorities and economically disadvantaged backgrounds 
enrolling into adult educational programs (Maralani, 2011). Since the 1990s, there has been an abundance of 
advocacy for Latinx education and career advancements (Arciniega, 2012). More attention has been focused on the 
education of the Latinx population due to the immigration of many to the United States. The Department of 
Education focuses on improving high school and college graduations with an increasing focus on the Latinx 
educational issues (Orchowski, 2010). The United States Department of Education awards federal grants to assist 
migrant students in obtaining their GED, enrolling in post-secondary education, and addressing specific needs of the 
Latinx population (U.S. Department, 2012). The Office of Migrant Education in the Department of Education 
oversees these grant funds and the programs awarded. 


As depicted in the chart below, the Hispanic (Latinx) population has higher dropout rates from school than 
blacks and whites. 


Figure 6 


Status High School Dropout Rates, 2000-2011 


(Share of 16-to 24-year-olds who are not enrolled in school and have not completed 
high school) 


40% 
35 
30 
25 
Hispanic 


20 


15 


NH White 


2000 2002 2004 2006 2008 2010 2011 


Notes: White and black exclude the Hispanic portion of those groups 


Source: National Center for Education Statistics The Condition of Education 2012 
Table 116 


PEW RESEARCH CENTER 


242 


The distribution chart below shows the levels of poverty in the United States by race. The Latinx (Latinos) 
are the second to highest percentages who are below poverty level. These poverty levels can be improved on by 
obtaining an education; however, these students typically need additional help to obtain their education and entering 
post-secondary educational programs. 


The Distribution of Poverty 
in the United States 


What are the most important conclusions you would reach from this figure? 


av 
D 
£ 
c 
re) 
o 
a 
¢ 20.6% 25-0 
— Be t Bicom co 8% 3% 
Ea om feel 
1 
Mam Percentage below Mmm Percentage Mm Percentage of 
poverty level of poor total population 


Need for Educational Technology in the HEP Classroom 


While the need for bilingual (English and Spanish) GED educational technology is nationwide, this 
presentation is on the experiences of the HEP program at a community college in the Southeast. All of the HEP 
programs are funded through the Department of Education: Office of Migrant Education. The majority of the farm 
worker students in the United States that HEP serves, according to OME are of Hispanic (Latinx) backgrounds 
(Department of Education, 2017). The Office of Migrant Education encourages the HEP programs to continually 
search for ways to improve all aspects of the programs, especially instructionally. Since the change of the GED 
from paper-based to computer-based, the HEP program has searched for any resources to assist in the instruction 
and educational practices. Majority of the HEP students have low educational levels and little computer skills. By 
finding and utilizing educational technologies in the GED/HEP classrooms, students are able to become more 
comfortable with the computers and prepare for the GED exam. There have been several English GED programs 
created to help the students to prepare for the computer-based GED exam. The greatest need is for more Spanish 
GED computer-based programs to help prepare this population for the GED exam. There are a few computer-based 
programs available; however, not all of them have a good Spanish translation. These types of programs confuse the 
students as the Spanish translation is not perfect. Spanish is a need in the HEP programs as it is the second most 
spoken language in the United States as depicted in the statistics below. 
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In addition to the technology use in the classroom, the HEP programs are continually exploring new ways 
to address the barriers to education for the migrant/Latinx population. This population has many barriers to 
education. HEP helps the students to overcome these barriers to become successful in obtaining their education. 
This study explores the benefits of incorporating technology into the migrant adult education program to assist 
students in overcoming these barriers to education, which are cultural, language, health, socio-cultural, childcare, 
working long hours, poverty, basic needs, and low academic educational levels. The barriers and needs inhibit the 
migrant adult students from obtaining their GED if not addressed. 


The HEP program provides academic assistance to the migrant adult education students. The migrant adult 
education students begin their educational journey by enrolling in an adult education program since most of them 


244 


were dropouts from the public school system (P-12). The HEP program serves the migrant and seasonal 
farmworkers. The program offers GED classes in English and Spanish. This assists the Latinx students in obtaining 
their GED in their native language. In addition, the migrant adult students need extra assistance when entering 
college. They need assistance in choosing a program they are eligible for (if the students are not United States 
Citizens, they cannot apply for state licensure, such as cosmetology or nursing). The Latinx students do not realize 
the power of education. They do not realize they can achieve their goals and desired careers. 


The HEP programs assist in bridging the gaps between the barriers and the access to education for the 
migrant adult students. The HEP programs are continually searching for ways to assist the migrant adult students in 
being successful in obtaining their GED. J. Gonzalez a former migrant student, former HEP director, and current 
Chief Operating Officer for East Coast Migrant Head Start Project states HEP programs provide a needed service 
that addresses a critical need and is important in rural areas where access to resources for non-English speakers are 
limited (Personal Communication, November 5, 2017). HEP is able to assist with transportation and childcare, 
which are some of the greatest needs with migrant adult students. The migrant adult students need to have educators 
that understand their specific needs and are willing to assist them. The HEP programs work with their local migrant 
programs to assist the migrant adult students further. The migrant programs help the migrant family and focus on 
the children. By collaborating, the two programs are able to assist the migrant families in being successful by 
addressing the barriers to education. 


OME focuses on improvements in the migrant adult education programs (HEP) to provide the Latinx 
population educational opportunities. The HEP programs focuses on the academic success of the Latinx students. 
The program’s goal is to assist students in obtaining GED credentials and moving forward with their education 
and/or career. As addressed above, the HEP program is able to assist with the financial barriers by providing limited 
assistance. The program is searching for technologies to use to help address the barriers of education for the Latinx 
students. 


Learning Theories and Motivation for Adult Learners 


Andragogy educational philosophy was an educational approach created by Malcolm Knowles to be adult- 
centered since adult learners have different needs than children which was based on the humanistic learning theory 
(Elias & Merriam, 2005, p. 13). Knowles recognized the barriers to education for adults and the differences in 
learning as compared to children. Knowles identified five assumptions of adult learners in andragogy. These are 
self-concept, past learning experience, readiness to learn, practical reasons to learn, and driven by internal 
motivation (Elias & Merriam, 2005, pp. 133-134). Adult learners do have different learning needs than the 
traditional P-12 age groups. The Latinx adult learners require more attention in education due to the barriers of 
education and special needs as mentioned previously. 


Technology in Education 


Technology is the application of knowledge in a specific subject (Merriam-Webster, 2018). Technology in 
adult education can be used in various ways, such as online programs, YouTube videos, Google hangout, Internet, 
applications for tablets, applications for phones, laptops, and Chromebooks. Since the changes of the GED exam in 
2014, there is a greater need for new methods of instruction and assistance for the students. The greatest challenge 
is the GED2014 changing to a computer-based exam. This results in students needing to be familiar with computers 
and online programs. Since the change of the GED to a computer-based format, there has been a decline in GED 
diploma attainments. According to the OME (2018), the GED diploma attainments were 74.5% in 2013, 2014 
66.6%, and after the GED change it dropped to 48% in 2015. This was due to the test changing to a computer-based 
format with a slightly higher level of difficulty. This change of the GED in 2014 created a greater need for 
technology usage in the adult education classroom, including the HEP classrooms. 


El Paso Community College is improving the access to education for Latinx students by creating 
educational classes with professors understanding the population, various learning options, and utilizing technology 
(“EPCC Serving,” 2016). Technology has various usages and can assist in the instruction of the Latinx students. 
Online instruction in GED and ESL classes would be an option in incorporating technology into the HEP programs. 
The HEP program is sampling a variety of programs to assist the GED and ESL students who cannot attend class 
every day. The goal is to utilize available technology for the students to access study materials and lessons at home 
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when they are unable to attend class physically. Since this population works long hours on the farms, they are 
limited on the amount of time spent in the classroom. Finding online instructional programs for the adult educators 
to utilize would assist the migrant adult students in studying and completing homework at their home. This would 
be a great benefit for the students; however, there is a limited amount of Spanish online programs. 


By using the tools available, the HEP programs are able to address the barriers of the students’ access to 
education. An online instructional program would help to address the childcare issue. Students would be able to 
complete majority of their studies at home and not worry about finding someone to take care of their child while in 
class. Technology could be incorporated into the classroom to access online resources on cultural awareness and 
learning about society. Also, by finding online resources and programs for the students to use at home, they are able 
to address some of the financial issues, such as transportation. Students are able to research their rights on working 
on the farms and domestic violence/abuse. In addition, students could research about preventative healthcare and 
learn how to better care for themselves and their families. Most of all, students could research educational 
opportunities and future careers with the Internet to help them decide on what they want for their future. 


Challenges Incorporating Technology 


Technology usage is becoming a need in adult education. The adult educational programs need to 
implement technology into the classroom to prepare students for future careers using computer skills. The issue 
with the incorporation of technology into the adult education classroom is the need for technological support for the 
adult education educators and staff (Gopalakrishnan, 2006). While some educators and staff are reluctant to change, 
the majority are willing to learn new technology to assist their students. To be successful, they need technological 
support. In addition, the students must be taken into consideration during this process of change. The biggest 
challenges found with incorporating technology in the migrant adult education programs were access to computers 
and the Internet. Several studies determined only 60% of families own computers and have Internet access and 
those who are considered minority and in poverty do not own a computer (Stern, 2006) 


Continuing Case Study of the HEP Programs 


The case study is continual at the institution. The task is to find a bilingual computer-based program to use in 
the GED/HEP classroom. This is a difficult task because the programs are expensive. Typically, an institution has 
to purchase a large number of seats to access the computer-based program to be able to use in the classroom. 
Another challenge in finding a good computer-based program for the GED/HEP classroom is to find one containing 
a proper Spanish translation for the Spanish-speaking students. Some topics to discuss in this presentation are: 


What types of educational technology programs are available for GED programs? 

Which educational technology programs are available bilingually (English and Spanish)? 

How do you gain access to sample the educational technology programs? 

How much time do you need to invest in sampling to determine the accuracy and importance of the 

program for use in the GED/HEP classroom? 

e How much time do you need to invest in sampling to determine if the Spanish translation is thorough for 
the HEP classroom? 

e Is it feasible to create your own educational technology programs for use in the GED/HEP classroom? 


Fellow Educational Technology leaders and instructors can use their expertise to give insights on the issue. 
Having a roundtable presentation can open up a discussion. All opinions and suggestions are welcome. In this 
presentation, the participants will be ask for any suggestions on improving the GED/HEP instruction, especially for 
the Spanish speakers. All input from the Educational Technology leaders will be helpful in finding a solution to 
improving the educational practices in the GED/HEP classroom. Input from other educational leaders will be 
valued. An intellectual discussion on the case study and looking for suggestions to find a solution to the problem is 
key. 
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Conclusion 


The number one objective of this presentation is to help students succeed. The goal in this presentation is 
to find a computer-based program that is available bilingually (English and Spanish) for the HEP students to access 
when outside of the classroom. Normal instruction will take place during the classes. The computer-based program 
is to be for additional study and for homework. GED/HEP students need more help outside of the classroom 
because most of these students are limited on attending classes due to working full-time jobs. The plan is to utilize a 
computer-based program to deliver extra tutorials, additional classwork, additional educational practice, timed 
testing practice, and improving computer skills. Each of these areas will assist the GED/HEP students in being more 
prepared and ultimately more successful in passing the GED exam to obtain their High School Equivalency 
Diploma. This will also prepare the student for their future career. 


Desired outcomes are to find solutions to the barriers to education for the migrant students. Technology 
offers assistance in addressing the educational barriers of the students. While there are issues to incorporating 
technology into the GED classroom, they are manageable. The main issues are access to computers, Internet, and 
Wi-Fi. Additionally, there has been little research conducted on the benefits of incorporating technology into the 
migrant adult education or HEP programs to address the needs and the barriers of education for the Latinx 
population. This study will focus on investigating improvements to migrant adult education (HEP programs). The 
desired improvements are student focused. The migrant adult education HEP programs want to see all of their 
students succeed in their academic endeavors. By finding solutions to the barriers of education, the students are able 
to be more successful. Technology has the potential to help to resolve many of the barriers to education. 
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Descriptors: Microlearning, Instructional design 


Introduction 


Microlearning refers to a learning strategy designed using a series of short segments of learning content and 
short activities that makes a microlearning module. It is also called bite-sized learning because it utilizes small, well 
planned, bite-sized chunks of units or activities (Hug, 2005). It is designed to suit the limits of the human brain with 
respect to its attention span and avoid cognitive overload, and research shows how effective it can be on student 
learning (Aitchanov, Satabaldiyev, & Latuta, 2013; Liu, Wei, Gao, 2016; Wang, 2017; Zhamanov & Zhamapor, 
2013). Although the concept of micro teaching existed for a long time, the term microlearning has not been used 
until a decade and a half ago (Hierdeis, 2007) and research that explores microlearning environments is still growing 
in the context of higher education. 


Literature Review 


According to Hug (2006), there are seven dimensions of microlearning: time, content, curriculum, form, 
process, meadiality, and learning type. They describe mainly the design aspect of microlearning. Although the 
design aspect is very important, but they lack the pedagogy and technology aspects, which are key elements to an 
effective microlearning design. Research conducted to investigate the opportunities of integrating technology to 
create microlearning environments is still at its infancy, and it focuses heavily on corporate training and adult 
learning. Research on higher education is very limited though. In higher education for example, a study by Liu, Z., 
Wei, L., Gao, X. (2016) found that 80% of college students took active part in teaching activities. They stated “it 
promotes multidimensional interaction and increases deep-levelled cooperation and understanding” (p.870). The 
researchers also found that microlearning environments inspired and improved the learning environment itself and 
students’ interest in learning. Zhamanov & Zhamapor (2013) started to implement microlearning technique in their 
university course and also received positive feedback from students. Students expresses higher level of interests to 
learn the subject matter, and the amount of materials learned has increased compared to previous years. 


Results from a study by Bruck, P. A., Motiwalla, L., & Foerster, F. (2012) with a university level course 
and two governmental training courses showed that learners had good satisfaction levels and high usage levels for 
the course materials. Similar results by Aitchanov, et. al. (2013), who examined the use of Twitter, a social media 
technology, in a microlearning technique for educational purposes. They collected date from college students 
enrolled in Advanced Programming in C++ course, and found that the majority of them enjoyed learning the course 
materials when it was delivered in small chunks using Twitter. However, Students suggested that they would like the 
number of tweets to be increased, and to implement this technique to learn other spheres. Kovachev, Cao, Klamma, 
& Jarke, (2011) explored bilingual vocabulary learning, and they found “promising results in enhanced flexibility in 
personal learning content creation and increased efficiency in filling knowledge gaps” (p. 51). Similarly, Wang 
(2017), investigated the effect of delivering the Engineering Mechanic Experiment content in short sequenced 
videos. As a result, the author reported that “the engineering mechanics experiment grade of undergraduate has 
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improved significantly, the service efficiency of mechanical equipment and degree of familiarity has improved 
sharply” (p. 130). The design of those short videos, however, resulted to limited student-student and student- 
instructor interaction. Interaction with the digital content in this case is the highest. 


In terms of evaluations, Giurgiu (2017) assessed whether students respond better to evaluation questions 
when they watch small segments of content followed by a number of evaluations, or when they watch large amounts 
of content with fewer evaluations. Findings suggest that smaller chunks of content helped students to better retain 
information and better perform in end-of-course test. Students who learned through microlearning technique took 
28% less time to answer their evaluations and did 20% better, took three times less to cover the course materials 
compared to students who did not. This suggests that trying to learn large content at once result to little interaction 
with content. 


Although research has shown its effectiveness, there is a lack of providing clear structure and theoretical 
framework for microlearning. This paper intends to fill the gap in research by providing a three step process of how 
to create a microlearning environment that help facilitate students’ retention of information. 


Step 1: Break down the content 


The first step in creating a microlearning environment is to break down content into smaller chunks. 
Microlearning is designed to suit the limits of the human brain with respect to its attention span and avoid cognitive 
overload, and it supports Hattie and Yates’s claim (2014) that learning new material/skills within several spaced 
short sessions is more effective than a single longer session. This does not mean that instructors cut down their class 
time but to cut down their class materials into mini ones. It is important at this stage to think about what are the 
must-to-know information and what are the ok-to-eliminate information when students interact with content. 


The following questions can help instructors when thinking about the “must to know” information for their 


classes: 
e@ What do I want my students to know and understand in order to move forward? 
e@ What are the 3 or 4 most important things I want my students to learn? 
e@ What are the most common mistakes students do that affect their learning and their grades? 
e What topics do I think that can be broken down into small pieces? 


These are just some example questions to ask for choosing appropriate content to break down. Once 
microlearning content is determined, it is essential to think about the pedagogical model to use and the design of the 
microlearning environment. 


Step 2: Time the Activities 


Another important elements to consider is the length of the learning activities when designing and creating 
an effective microlearning environment. Learning in small steps happens when activities are relatively short. The 
total amount to complete all segment of content in a microlearning module does not take more than 15 to 20 minutes 
for learners to complete. Within the microlearning module, micro activities are designed that takes learner 3 to 5 
minutes to complete. This can include 5-minute mini lecture, followed by 5-minute think-pair-share, etc. The 
purpose of this step is to revisit information multiple times through a variety of activities to reinforce and retain 
information and help it move to the long term memory. 


Step 3: Focus on a Single Learning Goal 

A Microlearning module focuses on a single learning goal students need to achieve. What makes 
microlearning effective is not just because an activity takes 5 minutes to complete, but also provide specific and 
targeted information. The purpose of this step is to ensure that learners take one step at a time to learn and meet the 


goal, before they move forward. 


When writing down the learning goal, it is important to first think about what is the one thing that students will learn 
by the end of the microlearning unit? Something they’ II be able to do, feel, or know as a result of interacting with 
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the microlearning content. For example, students will be able to compute basic statistical tests by the end of the unit. 
Once a learning goals for a lesson were determined, it is important to assign a single learning goa for each 
microlearning unit, and then think about its process to help students learn. Baumgartner (2013) discusses the theory 
behind microlearning and proposes a model of a competence spiral to scaffold students’ learning. In the first phase 
called Learning I, students absorb knowledge; this knowledge is basic and has no meaning yet (relates to 
behaviorism), In the second phase called Learning II, students acquire knowledge. Students in this phase interact 
with artificial environments and make their own experiences. Learning here in active with meaningful feedback is 
provided by the instructor (relates to cognitivism). In the third phase called Learning III, knowledge is constructed 
where instructors and students work together to master the course materials (relates to constructivism). In Learning 
I+, students proceed to higher level with more advanced concepts. Based on the Baumgartner’s model, Géschlberger 
(2016) proposed a social microlearning platform designed for all four phases. In Learning I, students create and 
share microlearning content. In Learning II, students evaluate, rate and improve the microlearning content. In 
Learning III, students are able to tag and collect content items. Learning I+, students interact with the microlearning 
content and solve low stakes quizzes, which they can take repeatedly to help them learn the materials. 


The Role of Technology 


Technology can play an important role in microlearning. It can be used to engage students with micro 
activities. The challenge with technology is that it is a fast-growing industry, sometimes it is hard for instructors to 
keep up, as they have teaching and non-teaching responsibilities. However, instructors often have knowledge about 
the content they are teaching, but not much how to integrate technology to help students learn. (Alqurashi, Gokbel, 
& Carbonara, 2016). 


Many of the technologies used in the classroom nowadays are have mobile friendly applications. In 
addition, the number of students who own mobile devices continue to increase. For this reason, digital microlearning 
environments should use mobile friendly applications, to allow learners to complete learning activities on their 
devices. The integration of mobile devices to create microlearning environments was discussed by Hug (2010). He 
emphasized on the importance of mobile devices because many reasons: (1) content displayed on mobile devices is 
usually a microcontent, (2) attention span and time periods are relatively short when presented on mobile devices, 
(3) a screen size in a mobile device is smaller than other devices, (4) mobile devices allow the design of micro-steps 
in formal and informal learning environments, (5) mobile devices allow the microlearning environment to be 
mobile, physical and social, and (6) finally mobile devices are often associated with micro platforms. 


Conclusion 


The microlearning model helps to ensure that students retain information. It does help usually distracted 
individuals to learn in short focused micro content. Instructors must also focus on the learning experience in the big 
picture when implementing microlearning. The application and the construction of knowledge that occur in class 
after students complete their microlearning content outside of class. By including all elements of microlearning (i.e. 
content, pedagogy, and technology), it can increase student engagement, enhance student satisfaction, and positively 
impact the learning experience. 
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Descriptors: Instructional Design Model & Cognitive Load 


The cognitive refraction model is a classification system for learning objectives that helps you understand 
the cognitive requirements necessary to achieve your learning goals. Learning objectives are grouped into 
conceptual bins based upon the work required to meet them, as estimated based on: the number of elements involved 
in the process, the choices that are to be made, and the concreteness of the information being presented. In the 
following presentation, I will address classifying learning objectives in the cognitive refraction model and using the 
classifications to guide instructional tool and techniques choices. 


Introduction to the Cognitive Refraction Model 


It is a complex endeavor to determine what instructional methods will make learning objectives accessible 
for learners. Great material is lost in droning lectures, ill-fitting reading assignments, and off-topic discussion 
questions. The design process is so difficult that some educators never undertake the challenge, instead accepting 
textbooks or premade curriculum that does not align with their learning goals. The cognitive refraction model 
supports the process of aligning instructional methods with instructor’s desired learning objectives. It classifies 
cognitive tasks into groups based on intrinsic load, the cognitive work inherent in conducting the task (Chandler & 
Sweller, 1991). Once learning objectives have been classified, it is easy to determine the activities and tools 
associated with improving learning in that category of load. 

The cognitive refraction model is an applied cognitive load organizing framework for instructional design. 
The basic premise of cognitive load theory is that learning is work and we have limited cognitive resources with 
which to perform it. When the cognitive load of a task is too great, learners’ work quality suffers (Chandler & 
Sweller, 1991). This can lead to incomplete understanding, difficulty applying concepts, and inability to determine 
how to proceed in problem-solving. Luckily, instructional tools can help with these issues; they can ease the process 
of content acquisition, provide practice in the procedures and skills associated with a competency, and support the 
development of self-monitoring skills. To share this information, I will discuss classification rules for the model, the 
work associated with each category and how to support its learning, and tools that can be used to support learning in 
a task or simplify the processing of a category. 


Intrinsic Load Based Classification 


The cognitive refraction model classifies information in a cognitive load framework, looking at the intrinsic 
load of the task to be performed. The definition of intrinsic load used with this model is more complex than 
traditional definitions. Past models of intrinsic load measured load by examining the number of elements of 
interactivity in the task--the elements being held in the mind and used for interaction during task completion. The 
evidence that elements of interactivity influence intrinsic cognitive load is robust (Sweller & Chandler, 1994). 

Elements of interactivity are a necessary measurement for intrinsic load because if the interacting elements 
are not processed together, the intended understandings will not develop (Sweller, 2010). For example, when 
considering possible factors affecting the velocity of a vehicle, attention needs to be paid to both the speed of the 
vehicle and the direction the vehicle is going. Looking at only one of those factors will not provide you with the 
velocity. Thus, velocity has at least three elements of interaction (speed, direction, and the interaction between the 
two). 

The number of elements of interactivity in a task is not absolute. The number is affected by expertise in a 
subject matter: experts will group chunks of information together to be processed, thus reducing the number of 
elements and reducing the cognitive load (Kalyuga, Ayres, Chandler, & Sweller, 2003). Other than this effect of 
expertise, many researchers consider intrinsic load to be fixed for a given task. I argue that there are other elements 
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that we can establish as influencing intrinsic cognitive load. For example, tasks related to the same content and 
elements of interactivity might have very different intrinsic loads when using different learning objectives (Sweller, 
2010). To refine the classic conceptualization of intrinsic load, I suggest two other factors that have been evidenced 
to have cognitive loads that relate intrinsically to the task. These are 1) choice/decision-making associated with the 
learning task and 2) loads of orienting information. 

The first additional factor in the model is decision-making. The literature does evidence the importance of 
decision-making in intrinsic cognitive load. For example, your accuracy reduces when the extraneous load is 
increased with interruptions for complex decision-making, but not for simple decisions (Merrienboer, Kester, & 
Paas). This is evidence that the decision carries a load. This is easy to evidence experientially as well. Imagine the 
cognitive load associated with listening to a story. Your intended goal is to understand the story, but not to make a 
decision regarding the learning content. At the end of the story, you are asked to take a pop-quiz on the story’s 
content. Now compare that with the load of listening to the same story after being told that you will be quizzed on it 
later. For each bit of information in the story, you must determine whether you need to put work into remembering 
it, and must invest your work where you decide it is useful for your score on the test. That second load is much 
greater. Even the same choice can carry a higher-cognitive load in different contextual situations: decisions that are 
“expensive”, in terms of resource use or distribution, lead to higher associated cognitive loads, including affect- 
related cognitive loads (Whitney & Hinson, 2009). 

The second additional load I suggest be incorporated is an orienting load. This load is associated with 
abstraction of conceptual information, mentally aligning concepts, and contextualization of information. Of these 
three load processes, abstraction is the most load inducing, so it provides the categorical rule for this load, even 
though, in actuality, all three orienting loads will influence an individual’s learning. Conceptual abstractness 
influences cognitive load related to the learning task in both visual (Brady, Konkle, & Alvarez, 2011) and written 
information (Crutch & Warrington, 2005). This is another intrinsic load that is easy to experientially evidence and 
logically intuit. The load of decision-making about a problem related to concrete information, such as estimating the 
percentage of room a table takes in the overall room, is obviously going to be less than the load of decision-making 
about an abstract problem, such as estimating the amount of freedom in a nation. It is harder to identify what 
information is important in evaluating abstract concepts, and, similarly, it is difficult to estimate these factors’ 
values. Thus, the cognitive refraction model categorizes concrete versus abstract cognitive processes in distinct 
categories of intrinsic load. Similarly, the load of mental alignment is an understood intrinsic load associated with 
the cognitive load involved in spatial rotation and map alignment (Tversky, 1993). If a map is oriented to the 
physical location of the map user, reading the map involves a lower cognitive load than if the user needs to mentally 
rotate the image to interpret it. Contextualization of information is also important, especially for experts (Grossman, 
2017). Embedding learning in its contextual application can reduce the work of determining how to apply it. This 
can be interpreted as making the abstract more concrete by simplifying understanding its application. 


The Cognitive Loads of the Learning Tasks 

The cognitive refraction model is a tool to categorize intrinsic load of tasks based on these three factors: 
elements of interactivity, choice/decision-making, and conceptual abstractness. The categorization rules have been 
simplified for you in a decision tree in Figure 1, and will be explicated here, as well. To categorize learning 
objectives, you begin by determining if the learning objectives involve decision-making. If the answer is no, that 
information is categorized as structure processing. This is the processing associated with conceptual understanding. 
The cognitive load associated with it is the work of keeping track of information, activating the right information, 
and inhibiting incorrect information. This load can be small with very narrow, simple topics or heavy with 
interconnected and contradicting information. 

If the answer is yes, there is some decision-making in the process, you must determine if the process 
requires only one decision or many decisions. If the processing requires only one decision, you next look at the 
outcome of the decision. Is this a concrete outcome, with a right or wrong answer, or is it a more abstract outcome, 
with multiple possible correct responses? If there is a right answer, this is categorized as action processing. The 
information from structure processing is still a cognitive load, but action processing adds the loads of identifying 
that a decision needs to be made and making the right decision. If the decision involves abstract information, the 
load is greater yet, with the additional loads of identifying important factors, estimating values of factors, and 
choosing a path. This is classified as heuristic processing. 

If you need to make multiple decisions, those decisions must be connected into a procedure that organizes 
them. That imposes a cognitive load as well. That load is different if the steps involve reasoning about a more 
concrete product rather than a more abstract product. If the end product is concrete, such as a paper or other material 
object, then, as you organize, you can compare to a conceptualization of the concrete end product. This is design 
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processing. It involves all of the loads associated with the decision-making on different steps of the process, as well 
as the additional load of connecting the product being designed to some concrete, constraining principles. The 
ability to check each step for accuracy limits the load of this category to some degree. 

The final category of processing is share processing. The work associated with share processing involves 
communicating information to other audiences. This work is sharing abstract information while comparing it to an 
abstract set of requirements, so the processing is very resource intensive. 
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Figure 1. Decision tree for the categorization of the cognitive refraction model. 
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Figure 2. The cognitive refraction model and the work associated with process categorizations. 
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This categorization process is diagrammed in figure one. The model itself can be seen in figure 2. These 
categorizations might seem abstract. To make them more concrete for you, let me provide a light-hearted example of 
categorizations related to learning objectives about big cats. 

* Structure: Knowing different types of big cats is an example of structure processing. Knowing 
characteristics of groups of big cats, like their coat patterns, is also structure processing. 

* Action: Being told to choose the tiger from a set of big cat pictures of different species and being able to 
identify and select the right picture is an example of action processing. 

* Heuristics: When shown a set of pictures of big cats with behavioral descriptions, choosing which big cat 
you would like to be stuck in a cage with for three hours is an example of heuristic processing. 

¢ Design: Designing an approach to your confinement with the big cat as to give you the best chance of 
health and survival is an example of design processing. 

¢ Share: Being told that your friend will be the one put in the cage with a big cat and that you need to share 
your design with your friend results in share processing. You have already come up with a great approach 
to staying alive; now you need to communicate this approach. 


There is another load that an applied cognitive science framework incorporates that is not a load in the 
cognitive refraction model; this is the load of metacognition. Metacognition is thinking about the processing 
involved with one or more of the previously identified categories: structure, action, heuristics, design, and share. 
This load is associated with planning future actions, monitoring current actions, and evaluating past actions. These 
are all very abstract processes that look at two abstract concepts as they change over time. These processes are 
cyclic and constantly recurring. You can add a metacognitive load to any other process category. This will increase 
the overall cognitive load, but helps improve the quality of the final product if there are enough processing resources 
for the task. 


Prescriptive Categorizations 


Different types of processing are supported by different tools. If you need to understand a category, a list of 
what is in that category supports this learning well. However, if you need to choose the best option out of a number 
of possible choices, a list only helps you remember what the options are; it does not help you weigh those choices. A 
decision tree would support this learning more. Each category of processing has tools and techniques that 
specifically aid those processes; what those tools are, and how they aid learning, was derived from the cognitive 
science literature. 

To align these tools with the model’s categories, I reviewed the cognitive load literature about tools and 
techniques that improved learning and categorized them based on the types of tasks they influenced. Methods that 
simplified learning in a situation or improved the learner’s accuracy were categorized by the learning tasks they 
could be applied to and then classified using the cognitive refraction model. From this work, I created a 
classification chart identifying the tools and techniques that can be used with that category of learning tasks. This 
allowed the model to be prescriptive: after categorizing the learning objectives, you can choose between the tools 
that support that category of learning. The logic behind the tool classification involved an analysis of the type of 
work each tool theoretically influenced and using its predicted influence for instructional design purposes. The 
following section goes through the explicating logic of tools and techniques associated with each category. 


Structure Processing Tools and Techniques 

Addressing contradictions: Addressing contradictions in material improves structure learning because 
overlooked contradictory information will lead to less accurate decision-making. Furthermore, if contradictory 
information is automated without being addressed, it can lead to ingrained decision-making biases and systematic 
misidentified or unidentified factors. 

Categorization offloading tools: Categorizing offloading tools are tools that make the process of 
categorizing easier. Examples include guide books and data classification charts. They simplify the process of 
identifying what fits into a schema and how it gets incorporated. 

Categorization practice: Practicing the process of categorization of information improves the automaticity 
of the schema information, helps identify areas of contradictions, and strengthens the chunking of schema 
information. 

Coherent structure: Having a coherent structure makes learning a schema easier because it provides an 
example of a successful way to model the schema. The learner can see the model created with the instructional 
structure and map their structure onto a similar framework. 
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Concept maps: Concept maps improve structure learning by making the important information explicit. 
They also reduce the work of schema organization through the spatial offloading of learning information into a 
graphical form. 

Contextual embedding: Having structure information embedded in the context the information will be used 
in simplifies the process of identifying interactions between the learning material and the context. This improves the 
usability of the material being learned. 

Embedding in prior knowledge: Schema information that is incorporated into previously learned materials 
is easier to learn because previously learned information is already incorporated into mental structures, allowing the 
new information to be attached to prior understandings instead of constructed from scratch (which requires more 
work). 

Reducing unnecessary information: Excluding unnecessary information from a presentation simplifies the 
process of schema creation. This is because the learner doesn't need to determine how to incorporate the unnecessary 
information into their understanding. 

Rule identification instructions: Information is often classified by rules that guide what information is 
placed in which category. Providing the learner with copy of these rules to be reviewed whenever needed reduces 
the work associated with the task of classifying information. 

Summaries: Summaries simplify the process of learning schemas because they share an expert’s 
perspective on the important material to cover in the schema. This simplifies the process of determining what is 
meaningful. It also increases learners’ accuracy in identifying important conceptual connections. 


Action Processing Tools and Techniques 

Checklists: Providing a checklist for action information simplifies the identification of needed steps and 
reduces the load of keeping track of the procedure being followed. 

Flow charts: Flow charts are another way of simplifying the identification of needed steps and reducing the 
load of tracking the procedure being followed. It does this in a spatial organization, which can further off-load the 
information for some learners. 

Just-in-time guidance: Providing just-in-time guidance during action learning improves learning by 
reducing the load of remembering the steps of the procedure and reduces the load of remembering the specific 
directions associated with the procedural learning. 

Modeling: Modeling action information simplifies learning because it shows the learner how the task can 
be successfully carried out in a contextually-embedded format. This reduces the work of identifying what needs to 
be considered. 

Role playing: Role-playing action information allows the learner practice the action in a lower-stakes 
setting and helps develops automaticity for aspects of the process. 

Timely feedback: Providing timely feedback helps with the learning of action information because it 
strengthens the incorporation of correct action and it enables the identification of incorrect action or contradictory 
action information. This improves the accuracy of future action practice. 

Video evaluations: By viewing a video recording of themselves performing an action, learners can self- 
evaluate their performance. This is particularly helpful when dealing with physical actions, where kinesthetic and 
spatial information guide the learning. 


Heuristic Processing Tools and Techniques 

Feedback: Providing feedback during heuristics processing allows the learner to recognize aspects of the 
process that they were doing well and to identify aspects that could use some improvement. It allows for the 
correction of errors and reduces the chance that errors will be regularly practiced and automated. 

Graphic organizers: Graphic organizers improve heuristic learning by reducing the work of mentally 
maintaining representations of multiple factors and their interactions simultaneously. Off-loading the factors to a 
visual modality reduces the working memory load. 

Guiding questions: Guiding questions direct the learner to the decisions and choices that must be made for 
heuristic processing. They guide the learner in identifying pertinent factors and interactions that influence the 
heuristic processing. This can simplify mental processing and make end products more accurate. 

Mental models: Mental models simplify the process of heuristics learning because they offload the work of 
representing multiple elements at the same time. This allows the decision-making process to be guided by the model 
and improves the accuracy of factor identification. 

Metacognitive skills training: Training learners in metacognitive skills (planning, monitoring, and 
evaluating processes) allows learners to strengthen their understanding of the decisions that need to be made and the 
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factors that need to be attended to. Metacognitive monitoring of heuristics can improve the accuracy of the decisions 
being made and allow for refining of the heuristic throughout the process. 

Outline: An outline is helpful for heuristic processing because it allows the learner to self-check that they 
are following the problem logic well and have identified the important aspects of a problem. The outline can signal 
important information, guide behavior, and provide a way to self-monitor learning progress. 

Partial practice: At times the cognitive load of a whole task is so large that learners quit instead of trying 
to surmount the work. In these situations, it can be helpful to break problems down into smaller parts and have 
learners practice part of the task before attempting the full task. This allows learners time to automate aspects of the 
task and chunk aspects of the schema. Partial practice allows the learners to simplify the learning task through 
experience before attempting the whole task. 

Process guides: Process guides direct the learner by breaking down the learning situation into steps and 
identifying the different steps in the process being learned. This reduces the work of deciding what to do, and in 
what order. 

Sharing rules of thumb: Rules of thumb are the guiding logic of decision-making that others rely on. By 
having experts share the considerations and estimations they use in heuristic processing, the learner is provided with 
expert guidance for making the same decision themselves. This reduces the work of deciding what to do and 
monitoring decision making. 

Worked examples: Providing worked example problems reduces the work associated with heuristics 
learning because examples show the steps needed and the information to include in order to solve a particular 
problem. This reduces the work of maintaining this information in the mind while processing. 


Design Processing Tools and Techniques 

Contextualization of information: Contextualization of information reduces the work of design processing 
because in design processing context can be very important for final design. If the context of the design is provided, 
the learner does not need to imagine or create a representation of the proper context. 

Feedback: Feedback reduces the work of design processing by identifying areas in the design or the design 
process where attention needs to be paid or errors have been made. It allows the designer confirmation that their 
internal representation of the process is similar to outside criteria. 

Graphic organizers: Graphic organizers reduce the work of the design process because they can organize 
work into steps, off-loading the working memory load required to hold multiple elements in mind at the same time. 
They can also store important structure information, such as where items are located or how systems are 
interconnected, so that the designer doesn’t need to be experts in those systems. 

Rubrics: Rubrics reduce the work of design processing because they make it easier for the designer to 
recognize important design constraints and reduce the load associated with evaluating whether the created product 
meets the desired specifications. 

Skill simplifying tools: Skill simplifying tools can reduce and improve the work of design work by making 
the quality of the design better, making the work easier, or both. For example, when designing a yearbook, having to 
cut physical pictures out is more work and harder work than using a digital design program (one you have the skills 
to use them). The caveat here is that the load of learning to use the tool needs to be low enough for the use of the 
tool to be possible. If learning to use the tool is too hard, then the tool will not be used, or its use will reduce the 
quality of the end product. 

Time management offloading tools: Time management offloading tools help organize time in larger 
projects to reduce the load associated with figuring out what to do, when. Setting a calendar that notifies you of an 
upcoming meeting, setting incremental due dates for portions of projects, and having a project timeline are all 
examples of time management offloading tools for design processing. 

Visual representations: Providing visual representations of the design product can reduce the work of 
design processing because it makes mentally representing the product in the mind easier. However, making the 
mental representation more concrete might also lead the learner to constrain their product and reduce creative or 
novel design processing. 


Share Processing Tools and Techniques 

Graphic organizers: Graphic organizers simplify and improve share processing by creating visual/spatial 
representations of share information. This does a number of things: 1) it gives the audience information in a different 
modality to reduce their load; 2) it supports the building of shared understanding through visual information 
organization; and 3) it creates a spatial representation of the information that can be referred to as a place holder for 
that information in information sharing. 
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Rubrics: Rubrics simplify share processing by explicitly providing evaluation criteria to evaluate the share 
processing. They highlight important information for the involved parties and simplify the process of determining 
what is necessary to share to create a shared understanding. 

Self-assessment practice: Self-assessment practice improves share processing by improving skills in 
determining how successful a sharing process was and helping to automate parts of the self-assessment process. 

Self-assessment guides: Including guides for self-assessment of the share processing can improve the work 
associated with sharing because it forces the learner to slow down and think about the process of information 
sharing. This improves the designing of future communications and helps the learner self-identify areas where work 
is needed. 

Shared communication tools: Shared communication tools reduce the load associated with share processing 
by making the sharing more powerful through the use of technological tools. An example of this is a webinar that 
includes visual and auditory information to ease the work of creating a shared understanding. 

Sharing guidelines: Sharing guidelines improve share processing by creating agreed-upon constraints on 
how collaboration and sharing are conducted. These guidelines allow the participants to simplify figuring out what 
to share and how to understand shared material. Guidelines might constrain what information is to be shared, the 
organization of the share processing, who is to be involved in the share process, or timelines associated with the 
sharing. Creating system constraints reduces the work of determining what to include. 

Templates: Templates simplify share processing by guiding the learner in what information to share and 
when to share it. The template provides structure and organization information for the share processing so that the 
process of determining how to create a similar sharing is simplified. 

Time management offloading tools: Time management offloading tools improve share processing by 
simplifying the work of having multiple parties share information, both synchronously and asynchronously. 


FACILITATOR TOOLS TO INFLUENCE LEARNING 
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Figure 3. A classification chart of the tools and techniques associated with each process category. 
Discussion 


The cognitive refraction model was created to align child trauma curriculum with professional mental 
health competencies. The project initially attempted to use Bloom’s taxonomy for the task (Krathwohl, 2002), but 
that proved to be not suitable for the work. Bloom’s taxonomy made predictions that did not align with a cognitive 
load framework on learning, and its predictions were not as directive as desired for a flexible training curriculum 
facilitated by mental health experts instead of educators. The cognitive refraction model was constructed as a part of 
a larger cognitive load framework that I call the Cognitive Ecocultural Framework. This framework uses 
instructional design concepts from problem-based learning (Barrows, 1986) and experiential learning (Kolb, 2014) 
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to shape instruction and learner/facilitator interactions in this collaborative, ecoculturally-embedded approach. The 
framework allows instructional designers to align their instruction with the realities of the learning setting. The 
model is part of this framework, but can be applied alone to determine instructional tools to support curriculum or to 
recommend successful techniques to use with your planned instruction. 

In this presentation I have provided the basics for getting started using the cognitive refraction model to 
categorize learning objectives and to select tools and techniques for instruction. The model can also be used to adapt 
existing curriculum for better learning or to determine appropriate assessment methods. While this instructional 
model was initially designed for mental health professionals around child traumatic stress, it is applicable across 
subject matters. Two of its most promising applications are to support trainings in other professional capacities and, 
as translated into classroom practice, to support next-generation science standards. There are many promising areas 
to explore and places to apply the model to improve learning. 

To validate this model, we are using design-based research instead of an experimental approach. We want 
to show its validity its usability. I am also presenting about a curriculum improvement project (Grossman & Layne, 
2018) and how we used heuristic graphic organizers to support complex problem-solving in a problem-based 
learning curriculum (Grossman & Layne, 2018). These studies both support the model’s usability and utility. 
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Descriptors: Collaborative Instructional Design & Problem Based Learning 


This presentation will discuss a collaborative project to improve a curriculum about child traumatic stress 
by integrating new instructional design principles with problem-based learning. This will include a discussion of the 
design process, how needs and constraints were included in the analysis, solutions to identified improvement goals, 
and the process used to create new curriculum tools. The objective of this presentation is to share experiences with 
other instructional designers conducting similar projects. 


The Core Curriculum on Childhood Trauma 


The Core Curriculum on Childhood Trauma (Core Curriculum) is a tool being developed by the National 
Child Traumatic Stress Network to educate professionals who work in different academic- and field-based child- 
serving systems to improve the national overall quality of trauma-informed care for children and families. Settings 
in which the curriculum is being implemented include community-based mental health clinics, teaching hospitals, 
academic institutions, juvenile justice systems, and child protective services. Curriculum trainees include practicing 
mental health professionals, such as social workers, psychiatrists, psychologists, and mental health counselors; 
mental health graduate students, interns, and post-doctoral fellows; and professionals who provide other services to 
trauma-exposed youth and families, including child welfare and juvenile justice system workers. The Curriculum 
uses Problem-Based Learning (PBL), a collaborative learning practice in which groups of learners are presented 
with complex problems resembling those encountered by practicing professionals. These problems contain multiple 
decision-making points that require learners to repeatedly sort through, integrate, and develop solutions for problems 
(Barrows, 1986). Our cases are segmented into pages focused on different aspects of a child’s case. A facilitator 
supports the group as they collect and consolidate information; guiding them through the learning experience. In its 
basic form, the Core Curriculum guides learners through a four-step PBL process comprising (1) identifying 
important facts from the case description, (2) formulating hypotheses about the case, (3) determining important 
considerations and next steps to take, and (4) identifying learning issues that require further research. 

The Core Curriculum instructional design team included a postdoctoral instructional designer, a clinical 
psychologist who serves as director of education in evidence-based practice at the Center, a child psychiatrist with 
expertise in childhood trauma and bereavement, a clinical care and implementation expert, and two social workers 
with expertise in childhood trauma and child service systems. The mental health professionals had all been working 
with the curriculum for a number of years, and were deeply invested in its success. 
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Design Process Overview 


At the beginning of this project, the stakeholders did not have a clear idea of what the project might entail. 
They had a strong belief that instructional design principles would positively influence learning from the 
curriculum--so much so that they had worked with an instructional design consultant for several years until they 
finally obtained the resources needed to hire a full-time postdoctoral fellow to assist them in realizing their 
ambitions. However, they had only a vague awareness of the areas of the curriculum that needed improvement. To 
identify the improvements in the Core Curriculum that were most needed, and to develop a strategy to sequence and 
carry them out, we chose to conduct the project as a form of design-based research. This is an approach in which 
products are designed to solve a problem in a specific context through iterative improvement cycles. It frames 
solutions as goals constrained by the systems in which the problems are embedded (Sandoval, & Bell, 2004). This 
particular collaborative model used a critical analysis aspect in the design process. This allowed content knowledge 
to be guided by experts in trauma-informed care, while the improvement process itself was guided by an expert in 
instructional design. 

The project began with an exploratory analysis of what should be included in the curriculum improvement 
plan. This was conducted as a critical needs and constraints assessment to determine what goals the current Core 
Curriculum was meeting and to identify the foundational aspects of the Core Curriculum that should be retained. 
After the assessment was concluded, its results were used to guide discussions of curriculum improvement goals. 
The primary aims identified were to support PBL facilitators, improve program implementation fidelity, create tools 
to improve critical reasoning and other learning objectives, and develop assessment materials for the curriculum. 
Many types of materials were needed to support these improvement goals. These included instructional guidance 
materials, case materials and resources, learning tools, and assessment tools. The existing curriculum materials were 
consolidated into these categories to determine which areas were well covered, as well as to identify gaps that still 
existed. Once we had identified a gap, we designed various tools and activities to address the learning needs. Once 
tools were created, they were distributed to our PBL facilitators for trial and feedback in an iterative process of 
refinement until the product was deemed adequate to fulfill the intended learning objectives. The design process we 
created to do this work is outlined in Table 1. 


Table 1: Design Process Steps 


1. Clarify the research question- What is the overall goal of the design project? 

2. Performa content needs and constraints analysis- What needs to be shared? What are the domains of 

information that need coverage? 

Summarize the content analysis results to aid in group communication. 

4. Collaboratively determine the smaller project goals and constraints that limit what can be done in the 
design process. 

5. Create a draft of design. In areas with uncertainty, leave unmapped areas for collaborative decision-making. 

Brainstorm possible design solutions to these areas, and use consensus decision-making to determine the 

courses of action. 

7. Create a completed rough draft of the product. 

8. Train collaborators and facilitators with the draft of the product, have them use it with learners. 

9. Collect feedback about the draft. 

10. Incorporate corrections and refinements derived from stakeholder feedback. 

11. Iteratively continue cycles of refinement until an acceptable draft has been created and formalized. 

12. Evaluate the final product’s utility and applicability, and the fidelity with which it can be implemented. 


LoS) 


2 


Needs and Constraints Assessment 


A conceptual content analysis was conducted on the Core Curriculum to identify the elements that made up 
the Core Curriculum, what stakeholders wished it to be, and what restrictions were necessary in the design process 
to retain foundational program features. Many materials were used in this needs and constraints assessment. The 
summary of these can be found in Table 2: Materials for Analysis. 
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Table 2: Materials Included in the Needs and Constraints Analysis. 


1) The process conducted to determine what the Core Curriculum was: 
e read through the curriculum materials 
e read articles about child trauma 
e reviewed past evaluation reports on the Core Curriculum 
e interviewed internal team members about their conceptual frameworks 
e interviewed facilitators about their facilitation practices 
e surveyed facilitators about how they applied the curriculum in their work 


2) The process used to determine what stakeholders wanted the Core Curriculum to be: 
e interviewed internal team members 
e surveyed facilitators regarding what they wanted the Core Curriculum to be 
e examined professional competencies about trauma-informed care 
e examined the contexts in which people wanted to use the Core Curriculum 


3) The process conducted to determine the constraints of the project: 

interviewed stakeholder about the learning goals of the Core Curriculum 
read through facilitator guides 

surveyed facilitators about their constraints in trainings 

reviewed past evaluation reports on the Core Curriculum 


Once this information was collected and consolidated, these areas of interest were used to construct rough 
models of each factor (what it was, what we wanted it to be, and systematic constraints) and to identify the primary 
components of each factor (e.g., clinicians had very little time for training, so trainings were constrained to being 
able to be conducted in a short time). This information was used to identify the improvement goals for the Core 
Curriculum. These improvement goals can be seen in Table 3 


Table 3: Identified Curriculum Improvement Goals 


1. Constraining all new additions to keep the rich, case-based approach of PBL. Everyone liked the existing 
cases. They wanted them retained, but with more materials to support their planning and use. 

2. Create short cases to introduce the core concepts within a more limited time frame, when needed. 

3. Create tools and systems to support PBL facilitation and fidelity of facilitation. Provide facilitators with 
guides for planning and instructing. 

4. Create tools to support critical reasoning, case conceptualization, and some of the more challenging 
foundational knowledge. The tool creation should be constrained to PBL formats to be used with Core 
Curriculum cases. 

5. Create ways to assess learning in the Core Curriculum, evaluate the Core Curriculum instruction, and 
improve program fidelity. 

6. Determine acceptable formats for Core Curriculum training to meet the learning goals. Planning was 
difficult for facilitators, which constrained the use of the curriculum. 


Building Towards the Identified Improvement Goals 


After we determined our improvement goals, we continued our design-based research approach to develop 
the curriculum materials to reach these goals. We created planning, learning, and assessment tools to supplement the 
existing Core Curriculum cases. One important factor in the Core Curriculum development was building in 
sufficient flexibility and support so that facilitators could adapt the Curriculum to provide trauma-informed 
education for different audiences across a variety of learning settings. For this to happen, we needed to identify 
general learning objectives for the entire Core Curriculum regardless of such considerations as the case used, the 
specific learning audience, or the setting. The general learning objectives were thus separated from the case-specific 
learning objectives that the curriculum had focused on previously. Those case-specific learning objectives focused 
on specific insights and principles covered by the case material and on idiosyncratic details of the case itself that do 
not generalize well beyond that situation. We needed general learning objectives that could cover the scope of what 
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we wanted the curriculum to cover, regardless of which specific cases were selected or which particular instructional 
tools and activities were used. 

The process of developing general learning objectives started as a content analysis similar to the needs 
assessment as previously described; competencies from different fields were coded and divided into categories. Key 
stakeholders, including the design team and Center leadership, were interviewed about the ideal content of general 
learning objectives and historical uses of the curriculum. This analysis was used to outline a prototype set of 
learning objectives. This prototype was then submitted to the senior management of the Center for review and 
approval. Management edited and re-wrote the learning objectives to frame the information in ways that they 
wanted. The design team then needed to convert the language of the general learning objectives to align with the 
intended curriculum assessment materials. The re-written learning objectives were iteratively cycled between the 
design team and senior management until a draft acceptable to both parties was produced. The final learning 
objectives can be found in Table 4. 


Table 4: General Learning Objectives of the Core Curriculum 


1. Apply the 12 Core Concepts as conceptual lenses to frame information and guide critical reasoning about a 
case study. 

2. Identify ecological factors hypothesized to influence children’s traumatic experiences and contribute to 
their post-traumatic adjustment. 

3. Incorporate relevant ecological factors into a case conceptualization, and use that framework to evaluate the 
hypothesized contributions of different case factors and guide case-related reasoning. 

4. Use critical reasoning to make judgments about the relative impact of various factors hypothesized to 
influence a child’s traumatic experience and post-traumatic adjustment. 

5. Clearly and accurately communicate appropriate trauma information to fellow professionals, clients, and 
family members within and across settings. 

6. Apply a trauma-informed conceptual lens to real-world aspects of professional practice, including 
assessment, case management, and treatment planning. 


Collaborative Instructional Design Process 


The instructional design process used to create curriculum materials was very similar to the design process 
of the overall project. It was a form of design-based research in which products were designed to solve specific 
problems in the curriculum. This tool design process can be seen in Figure | below. The design team created four 
categories of materials: 1) Instructional Guidance Materials, 2) Case Materials and Resources, 3) Learning Tools, 
and 4) Assessment Tools. Table 5 describes each category of materials. 
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Collaborative Product Design Process 


Figure 1. The collaborative product design process involved iterative cycles of prototype improvement. 


Table 5 Categories of Materials Constructed for the Curriculum Improvement Project 

Instructional Guidance — These materials guide facilitators in effectively using the Core Curriculum. This 
category includes materials to help learning facilitators plan their instruction, align 
their activities with their selected learning goals, incorporate appropriate materials 
that align with those objectives, and plan how to measure learning outcomes. 


Case Materials and These case materials are used in the PBL process. They describe a child trauma case 
Resources in sections so that group problem-solving can be conducted with the materials. Well- 
written case materials create rich, complex problem-based learning scenarios. 


Learning Tools Tools to support curriculum general learning objectives. They aid learners in 
communicating about the problem, consolidating information to create shared 
understandings. Also provide places to off-load information to simplify thinking 
about the problem-solving. Some learning tools also serve as assessment tools. 


Assessment Tools Tools to assess learning from trainings. There are both formative and summative 
assessment materials in multiple formats. Assessment methods are primarily 
performance assessment, to examine the complex thinking involved with case 
conceptualization and critical reasoning. 


For instructional guidance materials, we developed structured planning guides, organizing templates, and 
worksheets that aligned with the PBL cases. The purposes of these materials were to improve facilitator support in 
planning and increase instructional fidelity to the curriculum. The learning tools we created were sets of heuristic 
graphic organizers to improve collaborative conversations and deepen thinking to support our general learning 
objectives. These tools were designed to reduce the work of consolidating information and simplify the creation of a 
shared understanding. They allowed for more complex interactions between factors and an enriched shared 
conceptualization. 


The Instructional Approach 


The instructional approach used in this collaboration was a model constructed specifically for this work. 
Embedded in cognitive science research, specifically related to cognitive load, the Cognitive Refraction Model 
guides the selection of tools and techniques to support a learning situation. Instructional design concepts of Problem- 
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Based Learning (Barrows, 1986) and Experiential Learning (Kolb, 2014) shape the facilitation of activities in this 
collaborative, learner-centered approach, while Backwards Lesson Design (Wiggins & McTighe, 1998) 
conceptually contributes a sequencing for the model’s use. This cognitive load approach ensures that the materials 
created for instruction are scaffolded to the learners through facilitation techniques and instructional tools. The 
theoretical framework was constructed by reviewing and aligning applied cognitive load literature to derive a 
process-based categorization system organized by the work associated with the learning task. (Chandler & Sweller, 
1991). This model is also being presented on at this conference (Grossman, 2018). 


The Dissemination Process 


The first round of tools was created and introduced at an advanced training of our existing facilitators. 
These stakeholders were knowledgeable about the curriculum and aware of the five rich PBL cases that had been a 
part of the curriculum for the previous five years. The new tools introduced at the college included two short cases, a 
packet of 6 learning tools and activities to support general learning objectives, an experimental extended case with a 
large suggested activity kit, and some rudimentary planning tools. In hindsight, we introduced too many new 
materials at this training, which proved to be detrimental in the short term to the adoption of the new materials. 

After the initial training of the advanced facilitators, the project went into a lull. The majority of the design 
team attributed the lukewarm response to the materials to a deficit in the materials themselves. They deprioritized 
the project, and this led to bad communication and a slow feedback cycle. Furthermore, our initial collaborative 
approach had focused on development without an organized implementation plan. This lack of fore-thought 
developed into different understandings of the goals of the project and reduced the project’s productivity. 

Approximately a year into the project, this became a large enough issue that we took a step back and re- 
assessed the project and its forward progression. A report was created to summarize the work that had been done up 
until that point and to organize the project’s future directions. This report was shared with the design team and other 
key stakeholders. The report being shared and discussed improved communication amongst the design team. It 
allowed for the creation of timelines and a strategic plan for future implementation. This included a larger discussion 
on material prioritization, an introduction of new pathways of information sharing within the team, and structured 
reporting formats. Materials needed for the new facilitators were prioritized before cycling back to the advanced 
facilitators. This report and the new approaches taken afterwards renewed the design team’s dedication to the 
project. 

Another factor that renewed interest in the project was that when our facilitators used the new tools, their 
learners liked them. The feedback was positive enough that the facilitators who did implement the new tools became 
the tools’ champions. They used the tools, observed their role in their trainings, and came up with novel ways to 
implement them further. Since that time, many of these introduced tools have been refined based upon feedback 
from the advanced facilitators and redistributed in a more formal format. 

The design team continued to create products to support the curriculum. As the project continues to 
progress, we are developing more assessment tools and cycling back to the advanced facilitator materials. These 
materials include planning tools for creating new case studies, an activity kit to explore more complex skills 
associated with the general learning objectives, and instructional guidance materials for tool use. This work 
continues to build on the identified action goals at the beginning of the project exploration. 

To increase dissemination and usage of Core Curriculum material, we have formalized many of these 
products and posted them to a website accessible to our learning facilitators. Previously, facilitators had access to 
materials only at the training colleges or by requesting materials from the design team. By making the materials 
more accessible when our facilitators would like to use them, we hope that the materials will be used more, 
increasing program fidelity and learning. We have also been working to develop the public face of the curriculum 
through the National Center to increase awareness about the curriculum and further the dissemination of the work. 


Evaluation Study 


To better understand the learning associated with the instructional tools, we conducted an interview study 
about the tool use as observed by our advanced facilitators. These experts used the curriculum with the same cases 
both before and after the tools were introduced. We interviewed the facilitators about their tool use, the differences 
they observed in the curriculum’s instruction, and the benefits and drawbacks of the tool inclusion. Overall, 
facilitators liked and valued the tools. They reported that the tools allowed for discussions that incorporated more 
factors together, better collaborative sharing, and a more holistic understanding of the situation. Furthermore, they 
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appreciated the greater flexibility the tools offered with training options and the increased ability to assess learners’ 
current understanding. The tools increased engagement in the curriculum by breaking down the monotonous 4-step 
PBL cycle and allowed facilitators to tailor their lessons to achieve specific learning objectives, adjust to different 
audiences, or alter a training for the situation. The details of the study and the results of the interviews are being 
shared in another session in the conference, Heuristics to Improve Learning/Reasoning Skills (Grossman & Layne, 
2018) 


Collaborator’s Perspective 


This presentation about a collaborative process is being summarized and presented upon by an individual 
mind. This is theoretically misaligned with the collaborative approach. In order to buffer this limitation, this section 
includes the perspectives of the psychologist who served as the project’s primary investigator and the director of 
education in evidence-based practice at the Center. 

To date, the Core Curriculum has enjoyed good support across the Network. During past grant cycles, we 
focused largely on training new cohorts of PBL facilitators to assist in disseminating and implementing the 
Curriculum across the Network’s many sites, while conducting two relatively modest evaluation studies. Always in 
the background, however, have been larger questions about how the Curriculum fits in with and helps to address 
calls for competency-oriented education and training in core professional competencies, including traumatic stress. 
Our decision to hire a full-time post-doctoral fellow in instructional design represented a significant commitment on 
the part of the UCLA National Center and the Network to apply (and in some aspects, create and innovate) 
instructional design principles to a curriculum that is already enjoying broad dissemination—in perhaps as many as 
100 sites across the US thus far. 

We believe that the curriculum carries the potential of much more widespread dissemination and impact. 
This includes both “upstream” impacts, by changing how the next generation of mental health professionals is 
trained in foundational trauma-related knowledge, skills, and values; as well as “downstream” impacts, by 
developing effective, efficient, and engaging ways to “retrofit” the existing professional mental health workforce by 
furnishing them with in-service training in foundational (and advanced, as appropriate) competencies. The large 
scale of this endeavor (the Network now consists of 100 actively funded sites, with many more affiliate sites), its 
social importance (receiving strong bipartisan support in Congress), and the aim and structure of the Network have 
created a very unusual instructional design laboratory where products can be quickly developed, field tested, refined, 
and then broadly disseminated. Notwithstanding the setbacks described herein, the rapidity with which these steps 
have been taken reveals a much more rapid evolution in the development and dissemination of the Core Curriculum 
relative to past grant cycles. The creation of general learning objectives, case-specific learning objectives, and an 
assessment toolkit will position us to undertake a more rigorous evaluation of the Curriculum than has been 
heretofore been done. 


Discussion 


The purpose of sharing so much of our process from our curriculum improvement project is to share 
perspectives and experiences with other instructional designers working on similar projects. To support this work, 
we offer some of what we saw as the strengths of our process, the challenges we had to overcome, and the big ideas 
we feel might be valuable for other practitioners who are seeking to apply instructional design principles to other 
professional settings. 

Strengths of the Process. The biggest strength of the process was that our curriculum felt more complete 
and clearer as we made the improvements identified from our needs and constraints assessment. Our stakeholders 
said that they wanted specific changes to make the curriculum more powerful and more useable. The tools and 
materials that we created in our collaborative design process serve their intended purposes. Feedback from 
facilitators evidences the roles the materials are playing and the ways in which they are supporting learning. In this 
manner, we feel our collaborative design practice improved our accuracy in assessing the learning materials 
necessary and in designing the tools as well. 

Another focal strength of the collaborative design process is that it allowed us to leverage the expertise of a 
number of professional contributors. Our design team consisted primarily of experts in different fields of mental 
health, and our facilitator collaborators were also experts in different fields of mental health. This made their 
feedback on content knowledge particularly impactful and relevant to the instructional design. The primary 
responsibility of the instructional designer was to weave this information together in ways that were accessible to the 
learners. 
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Difficulties with the Process. The first difficulty associated with this collaborative design project was that 
members of the design team had multiple demands on their time and were not equally invested. Feedback on 
prototypes was postponed as other projects were prioritized. Additionally, the process needed to work with people 
with very different schedules. We found that having set check-in topics and spots allowed people to re-align back to 
what we were working on, but that those also seemed to make the process take much longer. In this manner we 
would re-analyze and re-make the same decision multiple times. We never found a more successful way around this 
challenge, and it continued to be a bottle-neck in the collaborative process. 

One of the strangest aspects of the Core Curriculum Improvement project was the translation across 
disciplines. The instructional design expert approached the curriculum design from a cognitive load perspective, 
focusing on improving collaborative load and scaffolding from facilitators to make learning materials more 
accessible to the audience. Materials were constructed with specific learning objectives in mind, and they influenced 
the end format of those materials. Evidence-based research was the foundation of the instructional approach, and 
each tool had a theoretical foundation for its construction. This intentionality proved difficult to communicate to the 
design team overall. 

While the tools functioned as intended, many members of the design team were slow to adopt new aspects 
of the curriculum. They had not used the new materials with learners, so they only started to understand the tools’ 
importance when their facilitator collaborators provided positive feedback. These collaborators became early 
adopters and championed the tool use integration. This created a cascading effect that slowly won over the more 
hesitant members of the design team. Nonetheless, the process of translating the cognitive science implications into 
easily accessible information for the design team was not as successful as it could have been, and that impacted the 
curriculum’s dissemination efficiency. 


Big Ideas 


Even given the difficulties with the collaborative process, key stakeholders felt that this work was very 
successful and are invested in future collaboration. Overall, we would recommend this process to another group as a 
powerful approach to collaborative decision-making and project design/implementation. In recommending a similar 
process, we would also like to provide some guiding suggestions derived from our experiences. 


1. Figure out your design and implementation process early on. When being invited to work on an 
interdisciplinary collaborative process, identify whether the people organizing the project have determined 
the methods they would like to implement to reach the project goals. If they do not have a process 
identified, as an instructional designer, you can improve your project by offering possible processes that 
might be reasonable. It is much easier to select from a few choices than to construct a process from an 
abstract theoretical framework. The model we created for this purpose was based in anthropology, 
ethnography, and qualitative research for problem exploration and iterative collaborative problem-solving 
for design purposes. 

2. One of the most important aspects in our collaborative process was the work of summarizing collected 
information so that the design team could make meaningful decisions about the material. This information 
should be in a written format so that team members can analyze multiple aspects of the project as they align 
with each other. Having the information written allows the team member to refresh their problem 
representation in the moment of decision making. Ideally, this summary should be broken down into 
segments of identified decision-making points. To get consensus between group members can be difficult, 
especially when interactions are often asynchronous and across a distance. Separating and demarking 
different choices to be made makes it is easier for team members to orient together for problem-solving. 

3. The instructional designer should identify ways of supporting team members in making valuable 
contributions. As the instructional designer, examine the processes your counterparts work well with and 
try to adapt your collaborative process to play on the strengths of your collaborators. For example, if a 
collaborator is much better at reviewing collected information than collecting the information, organize 
your collaborative process to have them incorporate their perspective later in the process. This allows 
people to contribute in ways that work for them and builds buy-in into the process. 

4. Iterative processes are good for collaborative endeavors because they allow for continual product 
refinement and they allow for team members to flexibly contribute when they have the time. 

5. In the creation of action goals, think of systematic constraints as well as the needs of a system. This makes 
it easier to prioritize what changes will give you the largest impacts for your time and effort. 
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6. Early adopters of tools and materials can field test and improve the curriculum. Moreover, their motivation 
can support generate interest and enthusiasm in program improvements. Identifying and supporting these 
individuals supports the project overall. 

7. As you progress on your project, remember to monitor your progress and re-assess whether you are still in- 
line with your original goals. Look at the products you are making and see where your goals need support. 
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Descriptors: Problem Based Learning & Cognitive Load 


This presentation reviews the creation of instructional tools to aid learning in a problem-based learning 
(PBL) curriculum about child traumatic stress. The tool creation was a part of a larger curriculum improvement 
project using instructional design principles to improve the use and dissemination of the curriculum. In order to 
improve the support of complex-problem solving related to child trauma cases, we created a number of heuristic 
graphic organizers to incorporate into our PBL structure. This presentation will describe the curriculum, explain our 
steps in tool development, outline our theoretical approach, and discuss the results of the process as seen through the 
lenses of our PBL facilitators. 


The Core Curriculum on Childhood Traumatic Stress 


The Core Curriculum on Childhood Traumatic Stress is a national curriculum used to educate mental health 
practitioners in different academic and field-based environments about child traumatic stress. The curriculum is 
being developed by the UCLA/Duke University National Center for Child Traumatic Stress in collaboration with its 
partners in the National Child Traumatic Stress Network (NCTSN). The aim of the curriculum is to support the 
mission of the NCTSN to improve the quality of trauma-informed care for children and families by raising the 
standard of education and training for their care providers in foundational principles of childhood traumatic stress. 

When the program improvement project began, the curriculum consisted of five detailed case studies 
written by a panel of experts in childhood traumatic stress. Each case focused on a fictionalized child and their 
family who had experienced a traumatic event or events. The cases were divided into sections, each ranging from 
one to two pages in length. Each section revealed new information about the child, the traumatic situation, and the 
context of the experience. These cases were conducted using traditional variations of a problem-based learning 
(PBL) model: Learners worked in groups of 6-12 students, reasoning section by section through the case. The cases 
were specifically designed to build complex problem-solving skills by intermixing both meaningful and less relevant 
information, thereby forcing learners to identify and piece together the relevant factors as they work their way 
through the case (Barrows, 1986; Hmelo-Silver, 2004). The curriculum used a medical variation of PBL, including a 
basic four-step PBL cycle. With each section, the participants work their way through the problem by (a) identifying 
facts, (b) creating hypotheses about the case, (c) determining next steps that align with the hypotheses, and (d) 
identifying learning issues where more information is necessary. This process is guided by an expert facilitator who 
helps to consolidate conceptual understandings and support the group process. (Layne et al., 2014). 

At the beginning of our project, we conducted a qualitative analysis to determine the aspects of the 
curriculum that were serving their purposes and the aspects that could be improved. This work revealed that PBL 
was an essential aspect of the curriculum that needed to be retained. Not only was PBL known in the literature for 
building evidence-based reasoning and prioritization skills (Savery & Duffy, 1995), but a survey of our facilitators 
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showed that it was well liked and considered a strength of the program. However, the survey also revealed that 
facilitators felt that the PBL cycle did not easily support all of the learning objectives, especially objectives that 
centered on solving complex problems involving multiple interrelated factors. 

Accordingly, one of the primary purposes of our program improvement project was to create ways to make 
this complex decision-making more accessible in the PBL framework. Our solution was to create graphic organizing 
tools that supported learners as they were working through these complex problems. The tool creation was guided 
by applied cognitive science, and the collaborative design process was guided by design-based research. After 
creating these tools, we invited the curriculum’s current facilitators to field-test them. We then conducted an 
interview study to explore the tool use and how it influenced learning from the curriculum. 


Design-Based Research Approach 


The curriculum improvement project began with a critical needs and constraints analysis to determine 
which goals the current curriculum was meeting and to identify foundational aspects of the curriculum that should 
be retained. More about this analysis is being shared at this conference in a presentation about conducting a 
collaborative program improvement project (Grossman & Layne, 2018). After concluding the analysis, we reviewed 
the results to determine our improvement goals. Our primary aims were to improve support for PBL facilitators, 
improve program implementation fidelity, create tools that could be used to improve clinical reasoning and other 
learning objectives using the established PBL framework, and create ways to assess learning outcomes. 

In order to create tools to support the identified goals of the curriculum, we needed to have a shared 
understanding of the curriculum’s learning objectives. Our needs and constraints analysis provided some of the 
most basic constraints of what should be incorporated within the curriculum. The curriculum was created to share 
foundational principles about child traumatic stress; these needed to be incorporated. Additionally, the PBL process 
had been selected because skills that it helped develop, such as case conceptualization and critical reasoning skills, 
were seen as necessary for this work. These learning objectives were easy to identify. To identify other relevant 
learning objectives that were not as apparent, we reviewed trauma competencies from different professional 
literatures, such as social work, psychology, and psychiatry. These were used to identify overlapping goals and 
prioritize the skills seen as necessary for trauma-informed practice with professionals. This information was 
summarized and used to create a prototype of the general learning objectives. Once they had been negotiated on an 
organizational level, the design team went to work making sure there were tools to support those learning objectives. 
To build these tools we used a design-based research approach with iterative collaborative design and product 
development, followed by a comparative product analysis to determine utility (Sandoval & Bell, 2004). We 
identified the gaps in the curriculum through surveying learning facilitators, trauma experts, and learners. We then 
designed a tool to bridge those gaps in our learning contexts. These tools were prototyped and then evaluated by 
facilitators as compared to the curriculum without the tools. 


Applied Cognitive Load and the Cognitive Refraction Model 


Cognitive load theory approaches improving learning through balancing the cognitive work associated with 
a learning task. If the load is manageable, the material can be accessed by the learners. However, if the load too 
large (or not large enough), learning will be diminished (Chandler & Sweller, 1991). The instructional design used 
in this project was an applied cognitive load model specifically created to support this work. We used the model, 
which we call the cognitive refraction model, to identify tools that might aid learning during specific learning tasks. 
It organizes learning objectives into categories of similar types of work to prescribe what type of tools would 
support learners in the learning task. The model can be seen in Figure | and is also being presented at this 
conference in a separate paper (Grossman, 2018) 
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The Cognitive Refraction Model 


Learning 
Objectives 


processing 
increases as 
more types of 
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Figure 1. The cognitive refraction model categorizes learning objectives based upon the cognitive work associated 
with completing the task accurately (Grossman, 2018). 


The model aligns these cognitive loads with tools that have been evidenced to support learning from the 
different tasks. We reviewed and categorized the cognitive load literature about tools and techniques that improved 
learning to align the tools with the model’s categories. Further, we organized methods that simplified learning in a 
situation, or improved the learner’s accuracy, according to the learning tasks where they could be applied. This 
allowed the model to be a prescriptive lesson planning tool: Specify your learning objectives, then use the model to 
categorize them and prescribe specific instructional tools to help learners achieve those objectives. 

The cognitive refraction model classifies the complex problem-solving we wanted to support as heuristic 
processing. Table 1 provides some of the techniques for simplifying the work of heuristic processing and for 
improving the accuracy of heuristic processing. 


Table 1: Examples of Techniques for Heuristic Processing 


Simplifying the work of Heuristic Processing 
* Providing guides to direct attention to important factors in the decision-making process 
¢ Sharing an expert’s guiding rules for decision-making before practicing the decision-making 
* — Providing tools that represent the information visually to reduce the work of multiple elements 
* Practicing the decision-making to make the factor identification and weighing-out more automated thus 
requiring less attention 


Improving Heuristic Processing 
¢ Providing tools to allow more factors to be compared simultaneously 
* Teaching self-evaluating skills to improve accuracy in decision-making 
¢ Incorporating feedback about the accuracy of the decision-making methods during future decision-making 
¢ Increasing contextualization of information into larger systems and frameworks. 


The cognitive loads associated with heuristic processing include the work related to identifying relevant 
factors for decision-making, estimating the importance of these factors, determining interactions between factors, 
predicting what possible outcomes might be, and selecting the best outcome out of the options. To design the tools 
to support these processes, we needed to know which tools might help with each type of work. The cognitive 
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refraction model’s facilitator tools classification chart laid out our options for us in this practice (Grossman, 2018). 
We identified how we could support the skills specific to our learning objectives. We decided to create a 
classification chart to help learners identify important factors in a case. Other tools provided conceptual bins for 
classifying factual evidence to help learners integrate case information and form hypotheses about how various case 
factors interrelate. 

Our heuristic processing was being conducted in a collaborative setting, and cognitive loads associated with 
collaboration would also influence learning from our PBL cases. The cognitive load literature most influential to this 
aspect of our work was the literature about collaborative load. Collaborative load is defined as the sum of the 
cognitive loads associated with collaboration on both the individual and the group level. Individual factors include 
the load of verbalizing thoughts and the load of maintaining an internal representation of the process. Group-level 
factors include both the load of creating a shared understanding, and the load of creating a representation of the 
group model of the problem-state (Kirschner, Paas, & Kirschner, 2009). We needed to keep these loads manageable 
for collaborative work to be successful. We decided to use graphic organizing tools to support these processes. 
Graphic organizers are helpful for collaborative heuristic processing in a number of ways: 


1) For internal representations: Graphic organizers help keep multiple concepts in the mind concurrently. We 
can only hold a limited number of factors in our minds at one time (Cowan, 2010); when this work is 
offloaded into a visual-spatial format, this load is reduced. 

2) For representations of the problem state: They organize information so that it can be processed more 
efficiently. By grouping information into meaningful conceptual bins, those categories can be processed 
together. 

3) For reducing the load of verbalization and the load of creating a shared understanding: It concretizes 
definitions to create a shared understanding. As collaborative processes create agreed-upon definitions and 
categorizations, it becomes more clear when a shared understanding has been achieved. 

4) For group representation of the problem-state: The visual representation supports the creation of a shared 
problem representation. Graphic tools can simplify defining and relating models in a collaborative process 
by predefining the factors in the model being applied. 


The Tool Development Process 


Our decision to guide learning with instructional tools was made after some tool construction had already 
taken place. One of the primary tools already existed (CHECKS/Double Checks Heuristic), and another had already 
been outlined (Seesaw Model). For the other tools, we used a collaborative design-based research approach to tool 
development, in which products were designed through iterative improvement cycles. This particular collaborative 
model included critical pedagogy, which allowed content knowledge to be guided by trauma-informed care experts 
while the overall improvement process was guided by an instructional design expert. The steps of the tool design 
process can be seen in Table 1. 


Table 1: Steps of the Tool Design Process 


1. Define the goal of the end product. 
Determine necessary content and scope 
Engage in collaborative decision-making about content of prototype 
Create draft of prototype 
As a design team, continue to create iterative drafts of the prototype until an acceptable draft is created 
Field test prototype with expert facilitators 
Collect feedback about the draft 
Incorporate corrections and refinements derived from the feedback 
Formalize final product 
. Evaluate the final product in relation to its use as a tool and its usability 
. Effect wide dissemination of final product 
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The Tools for the Core Curriculum 


The needs and constraints assessment identified three of the curriculum’s learning objectives as being 
particularly difficult to support: case factor identification, case conceptualization, and critical reasoning. These three 
learning tasks are all classified as heuristic processing in the cognitive refraction model. To support learning in these 
areas, we wanted graphic tools that could model the problem organization and simplify the work of creating shared 
understandings. We determined that we needed tools to serve three purposes: first, to help identify important 
ecological factors in a child’s ecocultural context; second, to aid in conceptualizing about the case and how 
identified factors play a role in a child’s traumatic experience and recovery; and third, to support critical reasoning 
around treatment prioritization and interventions related to a child’s case. For these purposes, we created four 
graphic organizing tools to be incorporated into the PBL curriculum: (1) Ecological Dimensions Categorizing Chart, 
(2) Protective Shields Factor Map, (3) Seesaw Model, and (4) CHECKS/Double Checks Heuristic. 

Ecological Dimensions Categorizing Chart (EDCC): The EDCC is our most basic tool. It helps novices 
learn to identify important case factors and classify them in meaningful conceptual bins. The tool presents learners 
with worked examples of ecological factors in a child trauma case. Learners are then asked to identify ecological 
factors in the problem case. Theoretically, it helps learners achieve a basic level of mastery in identifying and 
classifying case factors. 

Protective Shields Factor Map (Protective Shields): The Protective Shields is a more advanced tool to aid 
in identifying case factors, as it requires a basic working knowledge of ecological theory. Using this tool, learners 
identify case factors and classify them into ecological dimensions based upon whether they are theorized to serve as 
causal risk factors or as supportive factors. In a cognitive load theoretical framework, the diagram reduces the work 
of holding case information in the mind, categorizing case factors, and communicating about the interactions 
between factors. We hypothesized that this tool will serve as a learning tool, communication aid, assessment tool, 
and supervisorial tool. 

Seesaw Model: The Seesaw model takes a dichotomous (either-or) decision-making event and forces the 
learners to use evidence to support their decision-making. Using this tool, learners identify relevant case factors, 
weight each relevant factor according to its credibility, and then align the factors for versus against the decision. The 
seesaw model is hypothesized to be a good learning tool, communication aid, and assessment tool. Theoretically, 
this activity slows down thinking and requires explication of logic in a collaborative process. 

CHECKS/Double Checks Heuristics: The CHECKS and the Double Checks Heuristics are tools to organize 
case information for case conceptualization. These tools organize the information based on the role each factor is 
hypothesized to play in influencing the child’s experience and post-traumatic recovery. Using these tools, learners 
take identified relevant case factors and determine the role the factors play in the child’s case and how they relate to 
the possible case outcomes. The CHECKS Heuristic invites learners to sort case factors into four conceptual bins 
consisting of: causal risk factor, protective factor, vulnerability factor, and negative outcomes. The Double Checks 
Heuristic is a more advanced version of the same tool that includes four additional conceptual bins: positive 
outcomes, promotive factors, inhibitory factors, and facilitative factors. The CHECKS and Double Checks are 
hypothesized to be learning tools, communication aids, assessment tools, and supervisorial tools. We theorize that 
these tools categorize information into meaningful chunks and help retain multiple factors simultaneously. We thus 
hypothesize that the CHECKS and Double Checks serve as learning tools, communication aids, assessment tools, 
and supervisorial tools. 


Tool Evaluation Methodology 


To examine how the tools were perceived to influence learning, we turned to our advanced facilitators for 
feedback. Our advanced facilitators (N = 26) were trained in the original curriculum between 1-3 years prior to the 
tools introduction. These advanced facilitators were professionals in fields of mental health, such as psychology, 
psychiatry, and social work. The group contained clinic directors, university instructors, and community center 
representatives. They held a great deal of expertise in child trauma, but were not professional educators. Since their 
original training, these facilitators had using the curriculum at their work sites and occasionally at affiliated sites. 
This gave them experience in conducting the curriculum without the new tools. In June 2017, these facilitators went 
to our advanced college, where they were introduced to the new learning tools. 
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In order to gain advanced certification, participants were required to train in a case using at least one new 
tool. Advanced facilitators who had used one of these learning tools in their training were then candidates for 
participation in the study. If a facilitator was willing to participate, an appointment was set up for a semi-structured 
telephone interview about their tool use and their perceptions of its utility. These participants were part of our 
program and invested in the project of program improvement. They were considered collaborators more than study 
participants. 

Ten of these advanced facilitators agreed to contribute to evaluating their use of the tools. We interviewed 
them regarding the ways in which they were incorporating the new tools into PBL cases and the roles they might 
play in educating mental health professionals in foundational competencies. The interview included questions about 
their training particulars, the tools they used, details regarding how they incorporated the tools into their training, 
and how the tools were contributing to the quality of learning. We used this feedback for a number of purposes: to 
improve the tools themselves, to evaluate whether the tools were contributing to our learning goals, and to broaden 
the range of the tools potential applications. 


Feedback from Facilitators 


When we introduced the tools at the advanced facilitator training, the trainees appeared to be very 
circumspect, some even openly skeptical, in their reactions. We had groups of facilitators work through two of the 
new tools (the Protective Shields and Seesaw Model) in participatory exercises. We also introduced other two tools 
with the assurance that we would cover them later in a webinar. These instructional tools were only one part of a 
larger training, and the quantity of material they received during that two-day intensive training proved to be 
overwhelming for them. It was hard for the design team to tell if they considered the tools useful or enjoyable. 
Observing the training did inform us that we needed to clarify some of the tool instructions and to provide a 
terminology glossary for the language used in the tools. We analyzed and used this training feedback to refine the 
tools before formally distributing them. Whether or not they enjoyed the new tools, using at least one of the new 
tools was necessary for certification, so they were motivated to try. 

The phone interviews with the 10 facilitators revealed a good deal about the tools’ perceived utility. 
Everyone interviewed had used at least one tool in their training. Eight had co-facilitated the curriculum with 
another facilitator. This suggested that the early adopters were the ones who had the support of a partner. The tools 
were used in a variety of settings, including schools of social work, integrated health organizations, a special 
education collaborative, and for intern and clinical trainings. These trainings included shorter trainings (from 3-6 
hours), extended classroom settings (a 7-day intensive and course-work broken between 20 hours across a semester), 
and regular clinical work incorporating aspects of the program (for example, attending 2, 3-hour blocks for all 
interns). 

All five of our cases were used with the different tools, though more facilitators used the James case, a pre- 
teen case of domestic violence and child physical abuse, than any other case. All four of the tools were tried by at 
least one facilitator; EDDC, n=3; Protective Shields, n=7; Seesaw Model, n=3; CHECKS/Double Checks, n=4. The 
tools were appreciated by the facilitators who incorporated them. A primary appreciation of the tools was that 
facilitators valued how they changed the pace of the traditional PBL cycle, which could become repetitive. The tools 
were seen as giving facilitators more options in how they used their training time. Another common theme facilitator 
observed was the tools enriched group conversations about case factors, including deeper consideration about the 
specific roles each factor played, and the credibility of each piece of evidence. The tools also invited collaborative 
perspective sharing, leading to more contributed perspectives and pooled information. Many facilitators also 
remarked about the visual nature of the tools and how that was helpful for organizing information and comparing 
groups of information. The feedback about specific tools follow in Table 2. 


Table 2: Summary of Feedback about Specific Tools 


EDCC: Valued for giving novices structure to support conceptual understanding. It embedded the traumatic 
situation within the surrounding ecology in which it was happening. This tool was seen as very easy to incorporate. 
Protective Shields: Valued as a tool for helping hold and organize multiple perspectives. Facilitators particularly 
liked how it helped learners communicate about factors and identify areas where they needed more information. 
Dividing information into risk versus supportive factors allowed learners to focus more on supporting the child than 
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they normally would. It was seen as an intuitive way of organizing the information. Two of the facilitators organized 
the fact collecting in PBL using this tool throughout the whole case. 

Seesaw Model: Valued because it slowed down the learning process, forced learners to support their perspectives 
with evidence, and revealed personal biases. They particularly liked the conversations that came with trying to 
resolve differences of opinion related to case factors. This tool received some negative feedback, suggesting that 
time management was difficult with this too. Users reported that many professional decisions cannot be distilled 
down to dichotomous either/or options, and even seasoned professionals can reach very different conclusions about 
the same set of factors. 

CHECKS/Double Checks: Valued as a framework for formulating a working clinical theory, this tool was used for 
organizing information and structuring learners’ thinking about the roles factors play in traumatic situations. 
Facilitators who liked this tool applied it regularly and in multiple settings. It became a way of framing the 
reasoning around a case conceptualization and treatment planning. Users first used it as a communication aid and 
training tool, and then some expanded their application of the model itself by using it as a clinical supervision tool 
with their interns. The negative feedback about this tool was that the terminology of the tool needed pre-training, 
and that the Double Checks Heuristic (which contains 4 more factors) was too complex for beginning audiences. 


Tool adoption, dissemination, and implementation 


After facilitators had used the tools, they continued to use them. The tools were seen as engaging and 
especially valued for how they encouraged and facilitated perspective-sharing. They said they got more positive 
feedback from stakeholders as well. Each facilitator modified the tool use slightly to fit their situations; for example, 
some made hand-outs of the models for each learner, whereas others used the models with real-life case material in 
addition to Core Curriculum case studies. They became integrated into our facilitator’s professional practice. These 
facilitators share aspects of their work with each other during our regularly held consultation calls, which is helping 
to slowly broaden tool use in the advanced facilitator cohort. Based on this feedback, we created a number of new 
tools to support reasoning in other learning objectives. 

We also used facilitator feedback in another cycle of refining the curriculum. Because advanced facilitators 
judged the EDCC and the Protective Shields to be beneficial for all Core Curriculum learners, we chose to 
incorporate those two tools into the base curriculum and introduced them at our basic training for new facilitators. In 
this basic-level training, we were more intentional and direct in clarifying the flexibility that these new options 
created for facilitators. Introducing these tools at the basic training helped novice facilitators to conceptualize the 
tools as an integral and foundational part of the curriculum. Consequently, we observed that the novice facilitator 
cohort had an easier time incorporating the tools into their PBL facilitation than did the advanced facilitators. 

The feedback also helped us identify that the Seesaw Model could not be classified as a tool. The Seesaw 
Model was seen as a useful activity to help learners slow down their thinking, identify biases, and find evidence for 
beliefs. However, it didn’t flow well with the PBL cycle and wasn’t something that would be regularly incorporated 
in a section-by-section case analysis. Instead, we recognized that the Seesaw model falls into a new category of 
instructional materials: Activities. We have thus started to develop a Core Curriculum activity kit to support our 
learning objectives. We will design the activities in this toolkit to deepen thinking about case factors outside of the 
regular 4-step PBL cycle, thereby allowing learners to practice the skill once or twice during the case (such as at 
critical decision-making junctures as the case unfolds) instead of incorporating the tool into regular section-by- 
section case processing. The Seesaw Model will become part of the activity kit once it is completed. 


Discussion 


At the beginning of our work, our needs and constraints analysis revealed a need for better support for 
learning objectives associated with factor identification, case conceptualization, and critical reasoning. We 
hypothesized that we could support this learning with heuristic tools to balance cognitive loads during the PBL 
cycle. The tools we designed would theoretically support this learning by aiding communication about the case, 
organizing information into conceptual bins, making the information easier to retain, and simplifying the creation of 
a shared understanding. The feedback from our facilitators supported these hypotheses. Facilitators spoke of 
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improvements in organization of information, sharing perspectives, and combining information. The tools 
functioned as hypothesized and in doing so, improved learning outcomes from the curriculum. 

In the larger picture, this work supports using an applied cognitive load lens in instructional design. As 
anticipated, the practice of balancing the cognitive load of the learning task in PBL did improve learning from the 
curriculum. Facilitators spoke of holding deeper and richer conversations than they had previously had, and of 
building a deeper shared understanding of the material, than what was previously possible. We propose that mental 
health professionals require these diverse reasoning skills to handle the complex child trauma cases they encounter 
in their work. This curriculum invites them to practice that reasoning in a supported situation with low-risk decision 
making, given the obvious fact that making poor decisions with a fictional child carries far lower risks than making 
erroneous decisions in real-world cases. The use of the tools during learning aligns with and supports the 
classification skills and conceptual understandings that are necessary for performing this work in the field. 

In terms of increasing the applications of PBL in complex learning, this study supported the methodology, 
showing that heuristic tools could be used to scaffold learning of complex-problems in a collaborative manner that 
allowed for perspective sharing and improved reasoning skills. This was engaging curriculum that learners felt was 
easily applicable to the everyday practices of their professional work. 
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Introduction 


Social presence is an important metric for assessing learner performance in online learning environment. 
Asynchronous online discussions provide opportunities for learners to collaborate and socialize with their peers and 
instructors at their own pace. Brown, Collins, & Duguid (1989) asserted that people learn while interacting with 
each other through shared activities. But, Hewitt (2005) found that online learners consistently have limited 
discussion contributions in asynchronous discussions. Many researchers have examined the factors that caused low 
discussion participation and motivation (Deng & Tavares, 2013; Hew, Cheung, & Ng, 2010; Vonderwell & 
Zachariah, 2005). Besides learners’ personality traits and instructors’ pedagogical strategies, technical aspects have 
been identified as one factor affecting learner experience with asynchronous online discussions. 

A good interface design can help learners overcome the technical barriers that may occur while using a 
system (Metros & Hedberg, 2002). More importantly, a good interface design can positively influence the quality 
and the quantity of the interactions in an online learning community (Swan, 2004). Discussion platforms are the 
vehicle for online instructors to represent and conduct learning activities. If the discussion tools that online course 
designers selected did not meet students’ expectations, the activity might result in low participation rate and poor 
design. Compared to the limited participation in asynchronous online discussions, today’s learners spend significant 
time on social media interactions in their everyday life. The different perceptions with discussions in online classes 
and social media platforms may due to learners’ past experiences (Deng & Tavares, 2013). 

Many online learners have learning curves in online learning environments. The anxiety of using new 
technologies has created unnecessary obstacles to learner performance. Many asynchronous online discussion 
platforms are based on text. This text-heavy interface design leads to information and cognitive overload to online 
learners (Vonderwell & Zachariah, 2005). In this research, we examined the Mayer’s (2005) 12 principles of 
multimedia learning to provide design guidelines for online instructors. Therefore, instructors can search for the 
discussion platforms with the design features that can achieve their learning objectives. From learner perspective, 
the intuitive design of a user interface can help learners feel less intimidated and use the platform more. Thus, their 
learning performance can improve. 

Given asynchronous discussion activities are the primary means for online learners to interact with each 
other, it is essential for course designers to understand the nature of the discussion activities that can accomplish the 
learning objectives. A small body of literature studied learner perceptions of online discussion platforms by using 
the Technology Acceptance Model (TAM). However, this model only provides general information about 
technology by users (Liu, Chen, Sun, Wible, & Kuo, 2010). We have studied the design features of three online 
discussion platforms: Discussion Board (DB) in the Blackboard learning management system (LMS), Piazza, and 
Yellowdig. By interviewing the online instructors and observing their online asynchronous discussion activities, we 
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have created an interactive design matrix to help online instructors determine the appropriate types of discussion 
platforms for their online classes. 

The primary focus of this research is to identify the design patterns in designing effective asynchronous 
online discussions. Architect Christopher Alexander defined design patterns as reusable elements for solving 
recurring problems (Alexander, Ishikawa, & Silverstein, 1977). Design patterns are guidelines rather than 
prescriptions (Rohse & Anderson, 2006). In our research, the common and most useful design features in the online 
discussion platforms are identified as the design patterns for selecting asynchronous online discussion platforms. 
This interactive design matrix is built upon design principles and best practices to reveal the critical design features 
that fit online instructors’ needs for their intended asynchronous discussion activities. These design features can also 
tell online instructors the grading schemas should be included in their online discussion rubrics. 


Learning Experience and Intrinsic Motivation 


In this research, we proposed a design model and instrument to help online instructors create intuitive 
asynchronous discussion activities for learners. Besides the 12 principles of multimedia learning, we also integrated 
the universal design for learning (UDL) framework to guide our design. UDL has proved its positive influence in 
students’ academic gains and increased engagement (Rao & Meo, 2016). This framework focuses on the reduction 
of barriers in learning environment to create more inclusive classes for all learners (Al-Azawei, Serenelli, & 
Lundqvist, 2016). The traditional TAM mainly focuses on users’ perceptions on the usability of systems, which is 
not suitable for online learning contexts, as online learning involves hedonic aspects of human-computer interactions. 
Beyond the perceived usefulness and ease of use, a hedonic-motivation system adoption model (HMSAM) added 
“cognitive absorption” (CA) as a new factor impacting users’ behavioral intention to use the system (Lowry, Gaskin, 
Twyman, Hammer, & Roberts, 2013). The sub-constructs in CA were built upon the flow theory and intrinsic 
motivation. The finalized HMSAM revealed that users’ perceived use of ease affected their behavioral intentions to 
use a system through the mediation of perceived usefulness, curiosity, and joy (Lowry et al., 2013). 

Similarly, Jordan (2000) categorized the evaluation of a product design into three levels (from low to high): 
1- functionality, 2 — usability, and 3 — pleasure. This model suggests that beyond designing functional and effective 
products, the design of product should also bring pleasure to users. Pleasure refers to the satisfaction with the use of 
the product that users can emotionally relate to their real lives (Jordan, 2000). Some online discussion platforms 
have integrated the social and gaming mechanics in social network sites (SNSs), e.g., Facebook, Instagram, and 
Twitter, to their interface design (i.e., emoji, hashtag, leaderboard, likes, etc.), as many of online learners have used 
at least one SNS in their everyday life. Ryan and Deci (2000) asserted that intrinsic motivation leads to high-quality 
learning. Given that intrinsic motivation lies in people’s innate needs for competence, autonomy, and relatedness 
(Ryan & Deci, 2000), the similarities of the interface design between the technology tools used in online courses and 
learners’ daily life can trigger online learners’ intrinsic motivation. Hence, authentic asynchronous online 
discussions can effectively engage online learners. 

In the design and development process of asynchronous online discussions, instructors are also the 
stakeholders. By understanding the importance of pleasure in learning and self-determination theory (SDT), online 
instructors can create meaningful and engaging learning activities for learners. Moreover, online instructors need to 
monitor learner performance in discussion platforms to steer class discussions in the right direction. The discussion 
platforms they select must align with the HMSAM, which means that online instructors implement the platform with 
high intrinsic motivation. Compared to face-to-face classes, online instructors lack means for building connections 
with their students and understand their learners’ needs in a timely manner. If the selected online discussion 
platforms have built-in learning analytical mechanics, they will bring pleasure to online instructors. Therefore, 
online instructors can provide efficient feedback with their learners. The selection of an appropriate asynchronous 
online discussion platform should take the needs from both learners and instructors into consideration. 


Asynchronous Online Discussion Platforms and Themes 


Online instructors constantly face guideline dilemma in asynchronous online discussions, i.e., use of grades, 
use of number of posting guidelines, and instructor-facilitation (Hew et al., 2010). Our university has integrated 
three asynchronous discussion platforms in the Blackboard LMS. We have collaborated with university faculty 
members to design online and blended courses. Each of the discussion platforms can represent a distinctive need for 
asynchronous online discussions. From the conversations with the instructors and the observations of their online 
discussion activities, we have found different discussion themes as needed by the instructors on the three online 
discussion platforms. 
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1. LMS Built-in Discussion Platform 


Blackboard Discussion Board (DB) allows instructors to create independent discussion forums (see Figure 
1). Instructor can differentiate the activity requirements and grading schemas based on independent discussion 
forums. The interface design for learner interactions on DB is not quite intuitive, as it requires multiple steps for 
instructors and learners to view each student’s post. Instructors can only view the overall number of students’ posts 
in each discussion forum, while manually track conversation flows and the quality of their students’ posts. 
Instructors prefer to use this platform when they focus on content assessment, as well as need flexibility in grading 
different discussion forums. 
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Figure 1. An example of Discussion Board in the Blackboard LMS for asynchronous online discussions 


This discussion platform is suitable for end-of-chapter questions, as the purpose of this type of discussions 
is to assess each individual student’s understanding of the assigned topics. The focus is not on student interaction but 
on content comprehension. Rather than providing students’ opportunities for submitting individual reflections, a 
class discussion forum can help learners compare themselves with peers to check their understandings of the content. 

This type of discussion activity can enhance active learning, as each student can share coaching and 
scaffolding information with peers. Learners can improve their communication skills and deep thinking skills 
through active learning (Prince, 2004). Moreover, knowledge acquisition shifts from instructor-to-student to student- 
to-student delivery. We recommend online instructors to use these self-checking discussion forums as low-stake 
activities, as each online learner needs a process to disclose and polish their thoughts. Online instructors can turn on 
“participate before seeing others’ posts” function to give every student an opportunity to process and digest the 
learning content before seeing others’ opinions. 


2. Collaborative Learning Community 


Piazza is a wiki-style Q&A collaborative platform. Instructors can create discussion forums as separate 
folders in Piazza (see Figure 2). Learners can view most recent class interactions on the main dashboard without 
going into the specific forum. Unread posts, peers’ feedback, and instructor’s posts have different colored icons. The 
average answering time for a question and the class contribution leaderboards are available to the instructor and the 
learners. Many STEM instructors use Piazza to promote student collaboration, as students form a learning 
community when the instructor clearly sets up the discussion rubric beforehand. 
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Figure 2. An example of Piazza for asynchronous online discussions 


This type of discussion platform is suitable for learners to contribute answers and solutions to their peers’ 
questions. This type of discussion activities should be problem-solving oriented, as well as student autonomous 
discussions. The discussion purpose should not focus on the quality of each question but the quantity of 
contributions to peers’ confusions. We recommend instructors to use this type of discussion platform for extra points 
or no points for learners, as students should not be penalized if they do not have questions or not have time to 
answer peers’ questions. 

This discussion platform can help instructors identify the students who constantly struggle with the course 
content, as well as the proactive learners. Additionally, instructors can also find the concepts or homework that 
many students have issues. According to the UDL principles, instructors need to provide multiple means of 
representation for learners (CAST, 2018). Instructors can clarify the language and/or equations to scaffold online 
learners more effectively. 


3. Social Learning Platform 


Yellowdig is an online social learning platform with an interface like the SNSs. Learners can earn 
experience points (XPs) by posting, commenting, and like their peers’ posts. Each student can monitor their 
individual performance such as the quality of posts, class influence, and class popularity on the discussion platform 
dashboard. Yellowdig also has a built-in grading rubric assigning XPs by using the minimum word numbers of 
students’ posts and comments, which saves online instructors efforts in grading students’ works. The instructors we 
have interviewed indicated the quality and quantity of students’ posts in Yellowdig increased more than 50% 
compared to other online discussion platforms. Some instructors suggested Yellowdig was more suitable for 
augmenting in-class participations as in face-to-face classrooms rather than presenting “APA-styled formal 
reflections.” We found most instructors had a learning curve when they were firstly introduced to Yellowdig. 
However, many instructors felt their efforts were paid off after seeing student interactions significantly improve in 
their online classes. 
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Figure 3. An example of Yellowdig for asynchronous online discussions 


In the 2017 research report of ECAR study of undergraduate students and information technology, 42% 
undergraduate students rated social media as one of the tools that “wish instructors used LESS” in classes (Brooks & 
Pomerantz, 2017). Despite the popularity of social media tools being used in students’ everyday life, they do not feel 
comfortable to use personal social media accounts for academic works. This resistance of using social media tools in 
academic contexts provides opportunities for Yellowdig this type of discussion platforms, as they use the mechanics 
in social media platforms but adapt them into educational contexts. With the seamless integration into the LMS, 
learners do not need to register new user accounts with these platforms or worry if their personal life could be 
exposed to academic settings. 

This type of discussion platforms is suitable for class participation, which allows students to share outside 
classroom resources with peers. The layout of the discussion platform can promote learners to create a knowledge 
base for the entire class and utilize the shared resources in their future studies. Online students are encouraged to view 
and comment on peers’ posts, which can record participation points for students. However, the participation points 
that students accumulate in a social discussion platform should not be regarded as grades. Every student wants to earn 
100% in a grade, but their final grades can be affected by their deficiencies in certain rubric criteria. On the other hand, 
every student can earn 100% class participation points in a social discussion platform as the 100% participation is a 
baseline for learners to achieve, as long as they fulfill the minimum requirements, every student can be an “A” student 
in an online social discussion platform. The participation points also encourage students to contribute more to classes, 
as well as reduce online instructors’ workload in monitoring and grading student discussions. 


4. Design Patterns in the Discussion Platforms 

Besides the discussion themes we observed on the three types of asynchronous online discussion platforms, 
we compared the design features that the discussion platforms have, along with the built-in learning analytical 
mechanics they have. The design patterns used in the discussion platforms help us generate the design guidelines that 


online instructors can follow when they create discussion activities (See Table 1). 


Table 1. Comparisons of Design Features in the Discussion Platforms 


Platform | Comments | Forums | Hashtag | Badge Fae Posts | Subscription Upvotes* 
DB in 
Blackboard ‘a - x = - 
Piazza xX x xX xX xX xX xX 
Yellowdig x x x x x xX xX 


Note. *For example, likes, endorsements, and ratings. 
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Although some design features are common, their uses are varied. For example, students can subscribe to 
the discussion forums and posts in DB, but online instructors need to turn on and/or off that functionality in advance. 
Students have no options to make decisions. In contrast, both Piazza and Yellowdig can send push notifications to 
online learners unless learners unsubscribe the class discussion forums. Another example is the “upvotes” feature in 
the three platforms. In Blackboard, only instructors can decide whether to use this feature, whereas in Piazza and 
Yellowdig, the setting of this design feature is by default. Students can opt to not like or endorse others’ posts, but 
they will see the availability of this design feature. Additionally, instructors can change the points associated with 
“upvotes” in Yellowdig. 

We also examined the learning analytical mechanics in the three discussion platforms (See Table 2). 
Yellowdig has implemented all analytical mechanics and operate them automatically. In its dashboard, online 
instructors can view a summary of each learner’s behaviors. From learner perspective, they can view the total points 
they have accumulated in Yellowdig. Yellowdig automatically assigns participation points to learners once they 
reach the minimum words of posts and comments. The author of a post and a comment will also earn points if other 
students upvote their inputs. Moreover, online instructors can assign instructor badges or revoke points from online 
learners. 


Table 2. Built-in Analytical Tools in the Discussion Platforms 


Discussion Counts of Counts of Counts of Points Rubric Time 
Platform Comments Posts Visits Investment 

DB in 

Blackboard * a x a a 

Piazza xX xX xX xX 

Yellowdig x x x x xX xX 


DB in Blackboard has the second most analytical mechanics. However, it does not have an “all-in-one” 
dashboard to view student performance. Instructors can download class reporting to track student performance, such 
as numbers of views in each discussion forum, total counts of posts and comments in each discussion forum. 
Besides, online instructors need to go to grading center to manually assign grades to each student posts and 
comments. If instructors decide to use rubrics to grade student performance, they need to manually create criteria 
and level of achievements to embed rubrics into each discussion forum. This process usually costs a considerably 
large amount of workload for online instructors. This process also reinforces instructors to focus on the instructor-to- 
student delivery mode rather than a student-centric learning approach. 

Piazza does not implement every analytical mechanic to its platform due to the purpose of the use of this 
asynchronous discussion platform. Piazza is designated for class discussions of problem solutions, so the platform 
only records the number counts of online students’ questions and contributions. The grade book in Piazza is not 
integrated to LMS. Instructors can manually export the class report and insert them to the grade book in LMS. 


Design Frameworks 


Through the discussion themes and the use of design patterns in these asynchronous online discussion 
platforms, we aim to help online instructors identify the most suitable platform type for their discussion activities. 
We tie the principles and guidelines for discussion activity design with the specific design patterns that online 
instructors can implement into the discussion activities and learner assessment. We also identify the challenges that 
we could face in the design process. Although the advancement in technology has significantly influenced industries, 
its impacts in higher education are moderate, as many stakeholders have taken part in the process of new technology 
integration. If not being mandated by the institution, instructors create learning activities based on their teaching 
preferences. Specifically, they might choose different online discussion platforms for similar discussion activities 
due to their different comfort levels with technologies. However, each online discussion platform has its pros and 
cons; it is online instructors’ selected activity type and organization that affect learning effectiveness. 

Herrington (2006) posited that as many universities use LMSs to deliver online courses, instructors tend to 
focus more on information delivery through the use of a LMS than create meaningful activities for substantive 
learning. In this research, we created an interactive matrix to help online instructors identify the discussion platforms 
that most suitable for their desired learning activities and desired learning outcomes. We used the 12 principles of 
multimedia learning by Richard Mayer (2005) and the UDL framework by CAST (2018) to guide the design 
patterns that should be included in discussion activity design. We distinguished these two theories by emphasizing 
them on different aspects of creating asynchronous online discussion experience for learners. Multimedia learning 
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principles focus on the use of multimedia in discussion activities, whereas UDL principles focus on the narrative of 
discussion instructions. We examined the aspects that multimedia learning and UDL principles cover and overlap as 
shown in Table 3. We mapped these design guidelines with each discussion platform, which display in the 
interactive design matrix we developed in this research as a result for instructors’ intended discussion activities (See 
Appendix 1). 


Table 3. Comparisons between Multimedia Learning Principles and UDL Guidelines 


ies Content Organization Representation 
Graphic and narration (3, 
ar: Highlighted cues (2); 55 8;9): 
12 Multimedia Conversational than ape Corresponding words and 
; User-paced segments (6) é P g 
Learning formal (10) : : pictures are presented 
ke Clarify and decode 
Principles kel dieca near (4); 
symbols -and sext (7) No extraneous words, 
pictures, and sounds (1) 
Alternatives for auditory ead ; 
9 UDL or visual information (1); Ae ce Graphic and narration 
Guidelines Background Knowledge bola d aes (2) (1,5.1) 
3.1) 


Note. The numbers in this table are the principles or guideline numbers in the respective learning theory. 


Additionally, we followed 4 Maximums of Grice’s Cooperative Principles (1975): quantity, quality, 
relevance, and manner to group the essential discussion metrics that instructors can emphasize in evaluating their 
online discussion activities. After observing the analytical mechanics in the discussion platforms and instructors’ 
feedback, we added a new discussion metric time investment to the analytical mechanics can be used in online 
discussions (See Table 4). Many studies have shown that the amount of time students actively spent on tasks can 
affect their success in learning (Carroll, 1963; Lee, 2018; Wellman & Marcinkiewicz, 2004). The LMS and the 
asynchronous online discussion platforms have log files for time-on-task of students. Online instructors can 
visualize students’ frequencies of time-on-tasks in Piazza and Yellowdig; whereas they can export a class report 
from the LMS to analyze students’ frequency logs in DB. Adding time investment to the grading schemas for a 
discussion rubric can provide more information and perspective for online instructors to evaluate their student 
performance. The interactive design matrix we developed in this research displays key grading schemas for online 
instructors’ selected discussion activities (See Appendix 2). 


Table 4. Essential Discussion Metrics in Online Asynchronous Discussions 


Nee Content Expression Engagement 
Provable by Informative as required, 
4 Maxims of adequate evidence; no more or less; 
Communication Relevant to the Direct without 
discussion topic ambiguity 
Time Time spent in the learning 
Investment community 
Multiple media for 
UDL communication (5.1); 
Guidelines Multiple tools for construction 
and composition (5.2) 


Interactive Design Matrix for Online Discussion Activities 


This section introduces the development process of the interactive design matrix. We used PHP and 
JavaScript to create an online self-report survey for instructors to complete (See Appendix 1). Upon completion, this 
online survey generates a list of design guidelines and the suitable discussion platforms for instructors’ target 
students and discussion activities. Figure 4 shows the design model of our project. In our online instrument, 
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instructors will provide information on their target students’ characteristics, types of discussion activities, 
instructor’s technology skills, and needs for learner’s real-time data. Instructors will select answers in the format of 
multiple choices and Likert Scale questions. Based upon the responses to the survey questions, instructors will see a 
list of design guidelines that they can follow to build discussion activities, as well as a sample discussion rubric that 
can measure their desired learning objectives of discussion activities. We have included the design rationale for each 
survey question in the following section. 


Learner 
Characteristics 


Instructor’s Intuitive Types of 


Technology Online Discussion 
Sais Discussions Activities 


Learning 
Analytics 


Figure 4. The design model of creating intuitive online discussions 
1. Target student population 


Among the three types of discussion platforms, social learning platform is the most advanced platform for 
instructors to adopt. Depending on the student characteristics, traditional undergraduate students do not have 
limitations to any discussion platform if the discussion activities are meaningful. Graduate students tend to have 
higher motivation and needs for collaboration and in-depth discussions. Therefore, social learning platform is most 
suitable for them. For a mixed student population or non-traditional students, we recommend online instructors to 
use the LMS built-in discussion platform to conduct discussion activities, as these students tend to have longer 
learning curves. 


2. Motivations for Conducting Discussions 


We categorized instructors’ motivation for conducting asynchronous online discussions into three types. If 
the instructor plans to assess student understandings of the class concepts, they can conduct asynchronous 
discussions on any platform. If they plan to create a learning community to promote knowledge building, an open 
collaborative platform will suit their needs. If the instructor plans to provide opportunities for students to interact 
with each other, such as self-introduction prior to small group projects, a social learning platform will be an ideal 
interface for students to conduct this type of discussions. In addition, we have encountered some instructors 
implemented asynchronous discussions to their online courses only because it is a mandatory requirement from their 
departments or colleges. If this is the case, then a LMS built-in discussion platform can be an easy solution. 

This survey question also affects the grading schemas in discussion rubrics. If the instructor focuses on the 
content and institutional requirements, then quality and relation are the two major grading schemas should be 
included in the rubric. If the instructor emphasizes the development of a learning community or student interactions, 
then quantity and time investment would be the two grading schemas in the rubric. Besides, in a learning community, 
the relation of discussion posts also matters; while in student interactions, manner should be taken into 
consideration. 
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3. Current Problems in Online Discussions 


Instructors will indicate their primary concerns for their current asynchronous discussion activities. The 
aspects could be lack of interactions, lack of engagement, and lack of collaborations. Instructors can select all 
problems relevant to their current online discussions. If the instructor indicated that students lack interactions in 
current online classes, a social learning platform can help solve this problem. If the instructor considered student 
lack collaborations in the current online classes, an open collaborative discussion platform will be a good solution 
for this scenario. If the instructor indicated that the students lack engagement in online discussions, no specific 
discussion platform would significantly reverse this problem. But, instructors can follow the design guidelines we 
provide to refine their discussion instructions and activities. 


4. Types of Discussion Activities 


We identified five types of asynchronous online discussions. If the instructor decides to have students 
respond to a set of end-of-chapter questions, they can use the LMS built-in discussion platform. If the instructor 
wants online students to discuss homework or promote higher-order thinking skills, i1.e., debating and critiquing, 
they can use open collaborative discussion platform. If the instructor wants students to share out-of-class resources 
and contribute to open-ended questions, they can use a social learning platform to achieve their discussion goals. Or 
if the instructor wants to provide a platform for online learners to share group projects, they can use any of the three 
discussion platforms to conduct this type of discussion activity. 

The types of discussion activities influence the grading schemas in the rubric. For end-of-chapter reading 
discussions and group assignment sharing, quality and relation are the main foci in the rubric. For homework Q&A 
and higher-order thinking skills, guantity and time investment are the primary schemas in grading rubrics. If 
assessing student contributions to class participations, quantity, manner, and time investment are the suggested 
grading schemas in a rubric. 


5. Grading Schemas 


We also asked respondents for the grading schemas that they value the most. Respondents can choose from 
“quality of content’, “quality of interaction”, “numbers of contributions”, and “time investment” to reinforce the 
aspects that they want to assess learners. Each choice is mapped with one or more design metrics in learner 
evaluation. For example, if the instructor valued “quality of interaction” the most, then quality, manner, and relation 


will add more weights in the discussion rubric generated for the instructor once they complete the survey. 
6. Technology Integration Skills 


This Likert Scale survey question has a list of statements investigating instructors’ attitudes towards 
technology integration in teaching. Instructors will rate from 5-agree to 1-disagree for each statement to indicate 
their openness to technology integration. These technology integration questions are adapted from a free technology 
survey template in QuestionPro (https://labs.questionpro.com/a/q/questionpro--technology-survey-5543460). 
Respondents’ attitude towards each statement will add or subtract points they can earn for the openness to 
technology. The total points they have earned in this question are the indicators to low, intermediate, and high 
technology skills. LMS built-in discussion platforms requires low technology skills; open collaborative discussion 
platform needs intermediate technology skills; social learning platforms require instructors to have high technology 
skills and strong motivations. Additionally, if the instructor has equal scores for two and more discussion platforms, 
their points earned for this survey question will overrule the result. Since instructors will be the one that operate and 
manage online discussions, they need to feel comfortable about the platforms that discussions take place. 


7. “Must-Have” Features in Online Discussion Platforms 


This Likert Scale survey question is to investigate instructors’ needs for the features that they would like 
to see in online discussion platforms. This question consists of 5 statements, which are summarized upon the 
interviews we had with the instructors. Depending on the agreement levels (Agree, Somewhat Agree, Neutral, 
Somewhat Disagree, Disagree) the instructors rated, these statements indicate instructors’ preferences for different 
types of discussion platforms. Meanwhile, some statements also indicate the grading schemas that instructors want 
to use in the discussion rubric. 
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Figure 5 is a screenshot of a sample discussion platform and design guidelines that the online survey 
generates upon instructor completion. The bullet points are the key design guidelines that instructors can follow 
when they create the discussion activities on the suggested discussion platform. In addition, participating instructors 
also receives a sample rubric with weighted grading schemas that are related to their discussion activities. The 
instructors’ choices for each survey question as shown in Appendix 3 add scores to a specific type of discussion 
platform and/or grading schemas to their discussion activities. 


1. Which one is the best description of your students? 
Traditional undergraduate students 
Graduate students 
® Non-traditional college students 
Mix of student population 


2. Why do you want to have discussion activities in this online class? 
® Assess student understandings of the class concepts 
Create a learning community to promote knowledge building 
Provide opportunities for students to interact with each other 
My institute's requirements 


Question 


It is important for me to know 
how long students spend on their 
discussion activities, 


My online students should have 
multiple means to provide 
feedback to their peers. [Ex. Text 
comments, upvoting, voice 
comment, etc] 


It is important for my online 


Agree Somewhat Agree Neutral Somewhat Disagree Disagree 


students to view their peers’ level 
of engagement. [Ex numbers of 
posts and comments, time spent 
in total, offline or online status) 


3. What are the problems in your current online class? (Please select all that apply) 
# Lack of student interactions 
Lack of student engagement 
Lack of collaboration 
d It is important for my online 
students to have mobile apps. 


e _| like being able to intervene 
student participation in 
discussions. 


B Reset 


Recommended Discussion Platform 


You might consider using the built-in discussion board tool within the learning management system for this online class. 


When you write the discussion prompt for each discussion topic, please consider using words and pictures. If you decide to include any video clip to 
the discussion prompt, please make sure that the video clip is closed captioned. 

Please keep your discussion prompt conversational than formal. Your students will learn better from graphics and narrations than from animation and 
on-screen text. 


If your discussion prompt is longer than 3 sentences and including multiple tasks, please segment the information with bullet points or highlighted 
cues. 


Figure 5. A screenshot of the interactive design matrix for online discussion activities 
Future Steps 


We have demonstrated this tool to a small group of audience at our university. We will integrate this 
instrument to the online course development process in our department. Therefore, we can 1) better assist the faculty 
members to select and create intuitive asynchronous discussion activities; 2) evaluate the effectiveness of this 
instrument. We will add more background information questions to the instrument for us to collect data such as 
different academic disciplines and instructor demographic background. 
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Appendix 3: Interactive Design Matrix for Asynchronous Online Discussions 


1. 


Which one is the best description of your students? 
o Traditional undergraduate students 

o Graduate students 

o Non-traditional college students 

o ©Mix of student population 


Why do you want to have discussion activities in this online class? 
o Assess student understandings of the class concepts 

o Create a learning community to promote knowledge building 
o Provide opportunities for students to interact with each other 
o My institute’s requirements 


What are the problems in your current online class? (Please select all that apply) 
o Lack of student interactions 

o Lack of student engagement 

o Lack of collaboration 

o Not applicable 


What type of discussion activities do you plan to use for this online class? (Please select all that apply) 
o End of chapter reading discussion 

Class participation (e.g., open-ended questions, sharing resources) 

Homework Q&A 

Group assignment/project sharing and commenting 

Promote higher-order thinking skills (e.g., debate, critique and analyze topics) 


© “0 ©: ‘0 


Which grading scheme do you value the most when evaluating students’ discussions in this class? 

© Quality of content 

© Quality of interaction 

o Numbers of contribution (e.g., posting and commenting) 

o The time that students spend on the discussion platform (including reading posts and writing comments) 


Please select the answer that most accurately describes your feelings about using technology (5-agree, 4- 
somewhat agree, 3-neutral, 2-somewhat disagree, |-disagree) 

I feel confident in my ability to integrate multiple technologies into my instruction. 

I have a good variety of ideas and lessons for integrating technology into my teaching. 

The amount of time needed to prepare technology-based lessons deters me from creating them. 

I believe that integrating technology into my curriculum is important for student success. 

I am aware of the resources available by my institution that can help me learn how to integrate technology. 
I do not have the technology skills to support the students when they use technology for a project. 


O10 0. -O -O:- O 


Please select the answer that most accurately describes your feelings about using a discussion platform (5-agree, 

4-somewhat agree, 3-neutral, 2-somewhat disagree, 1-disagree) 

o It is important for me to know how long students spend on their discussion activities 

o My online students should have multiple means to provide feedback to their peers [Ex. Text comments, 
upvoting, voice comment, etc.] 

o It is important for my online students to view their peers’ level of engagement [Ex: numbers of posts and 
comments, time spent in total, offline or online status] 

o It is important for my online students to have mobile apps 

oI like being able to intervene student participation in discussions 
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Abstract 


Researchers have noted the critical need for developing teachers as designers, especially as it relates to 
designing purposeful instruction with technology. This study applied a problem-centered approach based on the First 
Principles of Instruction. The course’s goal was to develop preservice teachers’ application of TPACK when 
designing technology-integrated lessons. Using an embedded, quasi-experimental design to compare TPACK- 
application outcomes between face-to-face and flipped course implementations, preservice teachers’ growth was 
shown to be statistically significant in both the face-to-face and flipped course sections but not statistically different 
between sections. Additionally, a descriptive phenomenological analyses of preservice teachers’ experiences with 
the problem-centered principle provided additional explanations of these quantitative outcomes and offered insights 
for future problem-centered design. 


Introduction 


The terms instruction and teaching have been defined similarly by leaders in teacher education (TED) and 
instructional design and technology (IDT) (Elen, 2013). IDT leaders have defined instruction as “anything that is 
done purposefully to facilitate learning (Reigeluth & Carr-Chellman, 2009, p. 6)” while TED leaders have defined 
teaching as “the deliberate activity of increasing the probability that students will develop robust skill in and 
knowledge of the subject under study and coordinated within larger education aims (Ball & Forzani, 2009, p. 503).” 
Given the similar intents of these terms in their respective disciplines (i.e. facilitating learning and increasing 
students’ skill and knowledge) and their comparable methods (anything purposeful and deliberate activity), there has 
been great interest in developing teachers design thinking (Harris & Hofer, 2009; Koehler & Mishra, 2005). More 
recent research projects further exemplify the synergy as researchers have been interested in improving technology 
enhanced learning environments by engaging teachers’ in participatory design (Cober, Tan, Slotta, So, & K6nings, 
2015) and have also been examining how teachers can enhance their teaching by using instructional design 
processes for planning, designing activities, and selecting media (Carr-Chellman, 2016). 

Countering the development for teachers as designers are questions of necessity and purpose. Should and 
why do teachers need to be designers? One may argue that given most teachers’ time constraints, potentially limited 
expertise in instructional design, and likely few incentives for designing, engaging in design is not an advantageous 
endeavor (Kirschner, 2015). This argument is built upon the costs outweighing the benefits; the cost of teachers 
doing the work, the cost of developing their design thinking, and the cost of incentivizing these efforts. However, 
these concerns may be countered simply by the fact that teachers already are acting as designers (McKenney, Kali, 
Markauskaite, & Voogt, 2015). They are designers due to necessity and expectation. Institutional structures often 
need teachers to design and expect them to customize cookie cutter resources to meet the needs of their learners, 
their objectives, and their environment. Teachers also create and modify digital resources as an epiphenomenon of 
emergent technology adoptions by school districts. Especially for commercial digital tools not specifically 
developed for educational environments, teachers need to design for purposeful integration of the technology 
(Laurillard, 2012; Svihla, Reeve, Sagy, & Kali, 2015). Additionally, the push for deeper student learning has cast a 
fresh vision for the role of teachers; a vision that expects them to design powerful learning experiences (Martinez, 
McGrath, & Foster, 2016). The need and expectations for teachers to act as designers, however, does not negate the 
costs Kirschner listed (2015), but developing teachers’ design practices and thinking may increase the benefits, 
potentially decrease the cost, and thereby constitute a valuable area of research. 
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Purpose 


This study seeks to examine a problem-centered approach to developing preservice teachers’ design of 
technology-integrated lessons and resources. Further, this study will compare the impacts of the problem-centered 
approach when used in comparable face-to-face (F2F) and flipped course sections. Both course sections were 
designed according to the First Principles of Instruction (FPI), and both course’s learning outcomes were based upon 
the technological, pedagogical, content, knowledge (TPACK) framework (Koehler & Mishra, 2009; Merrill, 2002). 
While the broader mixed methods study also examined preservice teachers’ perceptions of their TPACK and 
learning experiences related to the entire FPI model, the next sections focus on explaining how participants 
interacted with the problem-centered principle of instruction as a means of informing the TPACK application 
outcomes. The research questions are the following: 

e How do preservice teachers’ application of TPACK to technology-integrated lessons compare 
between flipped and F2F sections of a course designed according to the FPI? 
e How do the preservice experiences with FPI’s problem-centered principle explain their application 


of TPACK? 


Perspectives 
TPACK 


The technological, pedagogical, content knowledge (TPACK) framework has significantly impacted 
technology-integrated design and preservice teacher development. As of 2014, over six hundred articles had been 
published based on TPACK, and over one hundred forty instruments had been devised to measure its constructs 
(Koehler, Mishra, Kereluik, Shin, & Graham, 2014). Scholars have applied the TPACK framework to reimagining 
practice in K-12 and higher education (Chai, Koh, & Tsai, 2013) and to reforming methods for developing 
preservice teachers (Lee & Kim, 2014). Prior to TPACK, similar frameworks for technology integration knowledge 
had been discussed (Angeli, Valanides, & Christodoulou, 2016). First introduced as TPCK, Mishra and Koehler 
(2008) changed TPCK to TPACK to emphasize the Total PACKage of knowledge needed for teachers to effectively 
design curriculum and instruction with technology. 


Figure | 
TPACK Framework 
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Note. Reprinted from “What is Technological Pedagogical Content Knowledge”, by Koehler, M., & Mishra, P., 
2009, Contemporary Issues in Technology and Teacher Education, 9(1), p. 63. 


293 


Extending Shulman’s PCK framework from three to seven domains of knowledge, TPACK is composed of 
three overlapping circles that represent three distinct domains (TK, PK, and CK). As illustrated by Figure 1, four 
hybrid domains (TPK, TCK, PCK, and TPACK) are formed from the overlapping circles. Technological knowledge 
(TK) refers to the knowledge of how to use software, hardware, and associated technologies. Pedagogical 
knowledge (PK) encompasses the knowledge of learning theories, instructional methods, and assessment (Chai et 
al., 2013). Content knowledge (CK) is the knowledge one has of a subject matter. The hybrid domains, 
consequently, transform isolated knowledge areas to create new forms of knowledge. As one example, technological 
content knowledge (TCK), is the knowledge of how to leverage technologies to research and create content within a 
subject area but does not consider knowledge for teaching. TPACK application, the course’s goal in this study, is the 
ability to select and integrate various technologies that facilitate instructional methods within a specific content area. 


First Principles of Instruction 


A problem-centered approach, based on the First Principles of Instruction (FPI), framed the design and 
implementation of this course. Merrill (2002) contends that applying FPI’s problem-centered principles of 
activation, demonstration, application, and integration will improve instruction’s efficiency, effectiveness, and 
engagement. The assumptions are that learning will be promoted when learner's prior knowledge is activated, or 
they are given opportunities to build this knowledge with new experiences. Learning will be promoted when new 
knowledge is demonstrated to the learners, and when learners apply this knowledge to varied problems and integrate 
it within their everyday experiences. All these phases are more effective when situated and connected within 
authentic and complex problems. Although the course in this study applied all components of the FPI, this paper 
focuses on the problem-centered principle, its corollaries, and learners’ related experiences and learning outcomes. 

In this study, the problem-centered principle engaged preservice teachers in solving an authentic and 
relevant problem through the iterative design of increasingly complex, technology-integrated, lesson plans and 
digital artifacts. The design problem, a technology-integrated lesson with supporting digital resources, was divided 
into five distinct phases. The number of phases was based partly on contextual factors such as the number of class 
meetings, but it was also related to components of a conventional lesson in childhood and early childhood education. 
The design phases were (1) analyzing the context, (2) selecting and aligning content and technology standards, (3) 
crafting learning objectives, (4) designing learning activities, and (5) creating assessments. 

Per the FPI’s show task and problem progression corollaries, each instructional module focused on a 
technology tool and a specific phase or component skill of the whole problem. The show task corollary posits that 
learners should be shown what problem or task they will be able to complete as a result of the instruction. Therefore, 
following instruction on the module’s new component skill, an entire problem or instructional scenario was 
presented to the students. For the problem progression corollary, the design problem shifted to a different context in 
each module, and its complexity increased as preservice teachers applied additional component skills. The increased 
complexity of subsequent problems and the provision of multiple problems intended to progressively improve 
learners’ skills. Although they began with fewer and simpler tasks within the problem, they gradually engaged more 
components of the design problem until they were assigned the entire problem. 

Since the initial component skill taught was assessment, students were only required to plan an assessment 
for the first design problem; all other components were provided. Learners then designed an assessment for each 
subsequent design problem. The second component skill, writing effective learning objectives, was provided in the 
initial design problem but required of students for the second design problem and beyond. This implementation of 
the problem progression corollary intended for students to develop mastery of isolated component skills situated 
within varied and authentic design problems. This proposed to scaffold them for their final design problem, when 
they would be expected to design a technology-integrated lesson and supporting digital resources for their field 
placement. 


Methods 


This study compared the impact of two FPI-based courses on preservice teachers’ application of TRACK 
when designing technology-integrated lessons within a problem-centered environment. To examine this impact, the 
quantitative data analysis focused on comparing the two conditions (F2F and flipped) on the TPACK outcome 
variables. Further, the study also explored preservice teacher learning experiences. The qualitative analysis of their 
reflection journals and the interview transcripts sought to better understand how they interacted with the content in 
each course version and how they perceived the FPI elements impacted their learning experiences. 
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During the spring 2017 and fall 2017 semesters, participants were recruited from four sections of a 
required technology integration course in their teacher preparation program. All participants were either elementary 
or early childhood majors with a focus on inclusive and special education. There were 32 participants total. Twenty 
participants completed the F2F course in the spring, and 12 completed the flipped course in the fall. 

Participants completed pre- and post- technology-integrated lesson designs using a specified template. Two 
raters individually applied the TPACK-based technology integration assessment rubric to each lesson (TIAR; Harris, 
Grandgenett, & Hofer, 2010). The TIAR consists of four domains, and scores can range from four to sixteen. 
Researchers met weekly to discuss scoring and reached consensus on lesson scores that differed by greater than two 
points. Finally, the primary researcher calculated gain scores from the differences between the pre- and post- scores 
and computed an ANOVA of gain scores following an exploratory analysis. 

As for the qualitative data, eight course assignments for each participant were analyzed, resulting in 
approximately 240 written artifacts. The researcher also interviewed four participants, three from the F2F course 
(Andrew, Aadan and Angie) and one from the flipped course (Brooke). Interviews ranged from 27 to 42 minutes. 
All interviews were recorded, imported into MAXQDA alongside participants’ reflections, and transcribed verbatim. 
A descriptive phenomenological approach to qualitative data was then applied to this analysis (Cilesiz, 2011). 

Phenomenological research attempts to represent the general nature of the phenomena by exploring it from 
the various perspectives of those who have experienced the phenomena (Matua & Van Der Wal, 2015). Thus, this 
study sought to represent the phenomena of experiencing a technology integration course intended for preservice 
teachers that has been designed according to the problem-centered principle of the FPI. The descriptive 
phenomenological method was selected for its alignment with the research questions focused on exploring 
participants’ experiences with course elements in both the F2F and flipped course versions. Further, a 
phenomenological approach has been applied to similar studies in educational technology to explore preservice and 
in-service teachers learning experiences and their TPACK development (Clark & Boyer, 2015, 2016; Lin, Groom, & 
Lin, 2013). 

In phenomenology, general themes or the essence of an experience are represented as a textural-structural 
synthesis. The essence represented by this study does not intend to be a universal truth for experiences in 
technology-integrated courses or experiences with a problem-centered principle. Rather, as Moustakas wrote, “The 
fundamental textural-structural synthesis represents the essences at a particular time and place from the vantage 
point of an individual researcher following an exhaustive imaginative and reflective study of the phenomenon 
(Moustakas, 1994, pp. 101—102).” As a researcher continues to study a phenomenon, an infinite number of 
experiences may be discovered (Husserl, 1931). The goal of the phenomenological analysis reported in the 
following sections was to best describe the general lived experiences of the participants in these problem-centered 
technology integration courses and the essential aspects of the experiences that related to their TPACK application. 


Results 
FPI’s Impact on TPACK Application 


While the ANOVA results showed no practical or significant differences between groups, an examination 
of the confidence intervals offered information about the potential statistical significance of each treatment group’s 
mean gain score. The 95% confidence intervals for this analysis did not include zero for either group. The intervals 
for each group ranged as follows: F2F group ranged from 2.61 to 5.59 and flipped group ranged from 2.30 to 6.70. 
These data show that while the two groups’ gains were not significantly different, both groups exhibited a 
significant increase in their application of TPACK to technology integration lessons as the mean gain score is 
statistically different from zero. 

To explore the statistically significant gains in TPACK application further, participants’ gains on the TIAR 
subscales were analyzed. As the TIAR subscales’ data did not meet the assumptions of normality for parametric 
tests, a Wilcoxon Signed Ranks test was applied. Table | displays the resulting pre- and post- medians, effect size, 
and degree of significance. All four domains of TPACK application as measured by the TIAR were statistically 
significant for both groups. These data indicate support for FPI’s potential to promote learning and indicate that the 
FPI may promote learning equally well in F2F and blended environments. Next, the descriptive phenomenological 
analyses of participants’ experiences with the problem-centered principle will be discussed and related to these 
quantitative outcomes. 
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Experiences with the FPI as Explanations for Growth in TPACK Application 


As displayed in Table 1, preservice teachers exhibited statistically significant growth in the application of TPACK to 
technology integrated lesson designs. Many shared that experiences with elements of the problem-centered principle 
positively influenced their learning, helped them apply knowledge in the course and provided an authentic context 
for their learning. Brenda clearly connected her problem-centered course experiences with learning how to integrate 
technology when she wrote, “Each week I feel I am becoming more and more comfortable with creating lessons 
with my design team, while incorporating the new types of technology we are learning each week and the ISTE 
standards. I feel that when the time comes around and I have to plan a lesson for my second grade students this 
semester, I will have so many new technology options that I can incorporate into my lesson.” 


Brenda identified the repeated opportunities for practice, her increased comfort with design and knowledge 
of technologies, and her confidence that these knowledge and skills would be useful. In terms of TPACK- 
application, Brenda’s statement aligned closely with elements of the TIAR. Preservice teachers exhibited consistent 
growth across all TIAR criteria with medians increasing one point. Preservice teachers’ experiences with the 
Table | 
Wilcoxon Signed Ranks Tests of Pre- and Post-TIAR Scores 


pale Mdn Mdn Sig. (2- Effect 
CRG Group i (Pre) (Post) A tailed) Sizer 
Ciaeiien F2F 20 2.00 3.00 2.86 .004 0.64 
Goals and Flipped 12 2.50 3.50 1.79 074 0.52 
Technologies Fipped@ 11 2.50 3.50 2.41 016 0.73 
iseheail F2F 20 2.00 3.25 3.47 001 0.78 
Strategies and Flipped 12 2.00 4.00 2.97 .003 0.86 
Technologies Fipped* 11 2.00 4.00 2.97 003 0.90 
F2F 20 2.00 3.00 3.03 002 0.68 

Technalogy  ~ srinosd 12 2.00 3.00 2.03 043 0.59 

Selection(s) ; 

Flipped* 11 2.00 3.00 2.68 007 0.81 

F2F 20 2.00 3.00 3.43 001 0.77 

"Fit" Flipped 12 2.00 3.00 2.56 010 0.74 
Flipped* 11 2.00 3.00 2.95 003 0.89 


*Outlier removed from this group based on a sensitivity analysis results. 


FPI-elements in the courses may offer explanations for the growth in these criteria and lay the groundwork for 
considering future design implications. The integrated findings discussed next will focus on relating participants’ 
experiences with two criteria from the TIAR, (1) Curriculum Goals and Technologies and (2) Technology 
Selection(s). 


Curriculum Goals and Technologies 


Emphasizing the alignment between technology selection and curricular goals, this area of TPACK 
application was noted as an area of concern during activation phase of instruction and early application phases. 
During an interview with Aadan, he reflected on his experiences with a one-to-one chromebook initiative, “Um I 
think technology can be a fantastic tool depending on how it's used.” He continued to describe how he often did not 
sense there was an alignment between technology and curricular goals when he was a student and found the tools 
distracting. Avery also noted times when technology seemed to be used for the sake of technology and not in service 
of content and learning when she wrote, “I would’ve rather done things my own way, but usually when technology 
was involved we were required to do things so specifically and include so many components (follow so many steps). 
Furthermore, the types of technology I used were very confusing to me at times and I feel that they were not helpful 
to my learning. I would’ve been better off without the technology and just doing things the old school way.” While 
some preservice teachers recalled learning important skills with technology, its’ integration in their K-12 education 
was frequently seen as superfluous to curricular goals. 
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In spite of these critical perspectives, preservice teachers at the beginning of the course still remained open 
to the possibilities as Bridget wrote, “The more technology a teacher can incorporate into the lesson the better. This 
will make [it] feel like a normal part of everyday life.” Bridget’s rationale for incorporating technology and her 
peers’ observations of technology integration attempts in their K-12 classrooms offer possible factors for the limited 
alignment between curriculum goals and technologies on the pre-course lesson designs. In preservice teachers’ prior 
technological experiences, content was often relegated, ignored, or misaligned with the digital tools’ affordances. As 
Bridget argues for more technology in the classroom, she does not support her position by relating it to learning and 
instruction. Her premise assumes that teachers should be concerned with technology becoming normative and that 
resulting instruction will improve. This perception of technology’s role does not account for content nor consider the 
role of pedagogy as technology becomes the end goal. 

Preservice teachers’ iterative lesson designs during the application phase of instruction may also explain 
the positive shift in their attention to the alignment of technology with content goals. As Arianna noted, “I would 
definitely say the repeated practices we complete in this class are productive to us as future educators. As it may be 
redundant and tedious for some, it still is eliciting great and continuous practice of effective ways to consider and 
implement technology into the classroom.” She and Angie both highlighted the benefits of this continuous practice. 
Participants’ experiences with the problem progression corollary were described as benefitting their ability to isolate 
and focus on key design elements. They were able to break down problems and build up their confidence. 

One of these key design elements, or what Merrill (2012) referred to as a subtask, was referenced in 
Bridget’s reflection on her group’s technology integrated lesson design near the end of the semester. Bridget wrote, 
“T think that it would be important to make sure that our objectives are closely aligned with the technology that we 
incorporated. I want the technology included to extend and build on the lesson, not be completely irrelevant or come 
across as busy work. I hope that as we finalize our lesson, we can nail down forms of technology that fit our 
objectives.” The continuous practice elicited by the problem progression corollary during the application phase of 
instruction provided a space to reconsider technology’s relationship to curricular goals within a lesson design. 
Bridget shifts from “the more technology...the better” to writing of the importance of alignment and fit between 
learning objectives and technology. Whereas considerations of learning and content were absent from her earliest 
reflections, she now wants the technology to “extend” and “build” and to be relevant. 

Further connecting the increase on this TIAR criterion with participants’ opportunities for practice and 
problem progression is yet another of Bridget’s reflections. She wrote, “I think that starting with most of the 
information filled in made me feel more successful in my abilities to plan a lesson that met both Common Core State 
Standards and ISTE standards.” In this statement she identifies technology and content standards, recognizes the 
need for lessons to address them, and relates the problem progression corollary to her sense of success and 
confidence. Brooke’s experiences also support this connection as she wrote, “By using this process I believe that it 
has benefited our groups understanding of what the most important components that go into a lesson are.” While she 
believed technology to be an important element to consider from the beginning, she wrote that should be considered 
alongside learning objectives and assessments. This iterative design process informed by the problem-centered 
principle and its corollaries was regarded as beneficial by the preservice teachers and may have contributed to their 
statistically significant increase in alignment of curricular goals and technologies. 


Technology Selection(s) 


The Technology Selection(s) criterion focused on the compatibility of selected technologies in the 
preservice teachers’ lesson design given their instructional strategies and curriculum goals. The previously discussed 
criterion evaluated alignment, but Technology Selection(s) evaluated the appropriateness of the selected 
technologies. While the median of participants’ technology selections in the pre-lesson designs corresponded with 
marginally appropriate technology selections, the median for their post-lesson technology selections was at the 
appropriate but not yet exemplary level. While they did not achieve the highest possible level of technology 
selection measured by the TIAR, both groups’ growth was statistically significant. 

When preservice teachers discussed learning about different forms of technology in this problem-centered 
environment, they described their experiences learning various digital technologies as a process: introduction to a 
technological tool (Activation and Demonstration), practice using the tool (Application), and evaluating its potential 
for a future lesson implementation (Integration). The activation phase introduced preservice teachers to new digital 
tools and structured experiences for them to build new knowledge about and with these tools. In an early reflection, 
Andrew acknowledged that prior to the course, he considered himself “tech savvy”. He then listed the many 
technologies he had already learned in the semester. Similarly, Aadan observed that course experiences exposed him 
to a range of technologies. Audrey and Alyssa tied these introductory experiences to preservice teachers’ capacity 
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for appropriate technology selections when they wrote that technology integration seemed more achievable after 
learning about new digital tools. An expanded repertoire of tools, Audrey also wrote, made technology selections 
more complicated. “It [is] difficult picking and choosing what technology to use (and potentially how to get the best 
use out of it) for a specific lesson/topic.” Yet, exposure to new tools was just the beginning of the process of 
learning about new forms of technology. 

Based on the FPI’s demonstration and application principles, preservice teachers observed modeling of 
procedures for using new tools. Along with this modeling, preservice teachers practiced using the tool with a partner 
or their design group. The following quote demonstrates the transition along this process. “For instance, I didn’t 
know that Google had so many different applications such as Google Slides, and through the two hours that I was in 
class I found out how to work the website through instruction and activities (April).” While April concedes a lack of 
knowledge of Google’s applications, she then acknowledges the benefit of working through instruction and activities 
in class to develop procedural knowledge. Constructing knowledge of the digital tool’s affordances through the 
construction of a meaningful artifact, “a Google Slides webpage”, was perceived by participants as critical for future 
technology integration decisions. 

Summarizing her experience with the process, Angie wrote, “I think now it is really impossible to plan a 
lesson without technology...it’s more about finding and using the RIGHT technology, not just finding any 
technology.” The TIAR makes the same assumptions as Angie; technology will be selected for the lesson. Angie 
concludes that technology selections will occur, and teachers must not settle for any technology. Through these 
processes and as indicated by the TIAR data, preservice teachers selected more appropriate technologies for their 
instructional scenarios at the end of the course, and participants’ experiences helped them learn the “ins and outs 
(Avery)” of several new tools as a means for improving their selections. 


Implications, Limitations, and Conclusion 


Several recent studies have applied the FPI to flipped courses. Some, as this study has done, compared the 
learning of F2F and flipped course designs. On the surface, Lo, Lie, and Hew’s (2018) results appear to contradict 
this study. In three of four disciplines studied, they reported that the flipped treatment group outperformed the F2F 
group. Critical to these results, though, was that instruction in the three F2F groups was primarily lecture-based. In 
the one flipped to F2F comparison that exhibited no differences in outcomes, the instructor stated that he embedded 
hands-on activities within the F2F course. Instead of countering the results of this study, these results rather seem to 
have a similar conclusion. The learning outcomes may be a result of the design of the learning environment. When 
designs included high leverage instructional practices such as hands-on learning espoused by the problem-centered 
principle, F2F and flipped approaches may prove equally effective. 

Preservice teachers’ experiences with the problem progression, task level, and problem progression 
corollaries and the robust TPACK-application outcomes further support the need to apply them as intended by the 
FPI. This is not to say that incorporating the FPI as described by Merrill is a simple process. Studies of the FPI 
application vary drastically in their application of the problem-centered principle, and a recent study of a flipped 
course based on the FPI even observed the absence of problem-centered activities during the physical class meeting 
(Lo et al., 2018). The essence of participants’ experiences in this study, however, indicate the importance of the 
problem-centered principle, its embedded corollaries, and proper implementation. 

A limitation to the results in this study was the variation of participants’ experiences in the field, 
particularly the mentor teacher’s TPACK, and how these contributed to the results. Nelson (2017) found that 
preservice teachers were impacted by the frequency and quality of their mentor teacher’s integration of technology. 
Technological competency was not considered as mentor teachers were selected, therefore, this variable was not 
measured. Knowing this could potentially impact preservice teachers’ implementation of the lessons factored into 
the decision to isolate planning as an outcome. First order barriers such as access and mentor teachers’ attitudes 
toward technology were evident in preservice teachers’ reflections and could not be controlled. The TIAR was 
purposefully applied to measure TPACK application as it was intentionally constructed neutral of a constructivist 
orientation and student-centered uses of technology. Its main focus was the alignment of components within a 
technology-integrated plan for purposeful integration and not evaluating the plan from a theoretical perspective. 
Applying the TIAR was intended to reduce the impact a mentor teacher might have on the score, as it was believed 
preservice teachers could score a maximum possible score even with first-order barriers present in the placement 
context. 

The knowledge and skills for designing digital resources and technology-integrated lessons is often of 
practical significance for teachers. Emerging technologies regularly require teachers to design their own resources 
and activities for technology to be integrated purposefully (Laurillard, 2012). Exploring design practices for 
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preservice teachers represents a synergistic agenda for teacher educators, teachers, and design researchers (Svihla et 
al., 2015), and a problem-centered approach based on the FPI appears to be a promising method for structuring the 
development of these practices. 
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Abstract 

Educational technologist performs as a catalyzer in supporting active and effective learning in an educational 
system. In Thailand, Educational technologists’ roles are delineated in the Chapter IX of 1999 National Education 
Act 1999 that were practically media profession who analyze, design, develop, and deliver media in a teaching and 
learning system such as radio, television, text book, and printed media, as well as all types of communication 
technology, while using research as a tool in the process, and practical professional development as well. None of the 
study has been done to solid instructional technology standards and its proficiency in Thailand. To strengthen the field 
of instructional technology, the study is aimed to research standards and competency for Thai instructional 
technologists. 

The study was employed by systematic reviews of current research and existing competency standards of 
educational technology after year 2010, coupled with an in-depth interview with experts in educational technology 
filed. The major standards of revision were retrieved from mainly organizations in the United States including: 
Association for Educational Communications and Technology: (AECT standards, 2012), the International Society for 
Technology in Education (ISTE standards, 2016), and including such as ISTE Standards for students (2016), and 
including major standards modified on the basis of ISTE standards namely the Washington State K-12 Educational 
Technology Standards, and Michigan Educational Technology Standards. 

The study was found the proposed educational technology standards including six areas: |.profound 
knowledge in technology and its adaptation to education, 2. system design, 3.professional development, 4.learning 
environment, 5. research and ethics, 6.localized and globalization. The proposed standards should be further developed 
and will be a potential countrywide implementation. 


Keywords: educational technologists, educational technology standards, Thailand 


Introduction 


The rapidly changing world of today's transition to the digital age has had an impact on human society in all areas, so 
that we can adapt to those changes in order to survive sustainably. It can be said that the advancement of technology 
affects the work system in all professions, which face the challenge of change. Like the educational technology 
profession. Educational technologists are one of the top professions that play a role in the development of education. 
This is to facilitate and support the educational system and teaching for teachers to improve the learning of students 
to be effective. 

The literature review found that educational technology standards were not developed or improved from the 
main organizations of the educational technology profession. It is only the research of graduate and doctoral students 
who develop the standards and competencies of educational technologists. There are a few and over 10 years of 
research that show the lack of continuous development and the lack of awareness of the importance of the profession 
that affects the role of educational technologists in the society. This has led to the development of standards of 
educational technologists. This problem has been the same as many educational technologists in many countries. In 
fact, educational technology plays an important role in supporting the development of change in the way that the 
educational system is good. On the other hand, they often have less role in society. This is due to the lack of a clear 
professional structure (Fox and Summer, 2014), or it can be said that this is a concern for the overlap of educational 
technology roles. (Browne and Beetham, 2010) This concern has occurred with broader educational technology. 
Research has reported that the challenges of 21‘ century technology educators affected by the advances in technology. 
How will they guide their educational goals? In addition, the development of the technology used in the teaching and 
learning of teachers to improve the knowledge and skills of students (Mayes, Natividad, and Spector, 2015). From 
this problem, the study is aimed to research standards and competency for Thai instructional technologists. 

The development of educational technology standards is a positive impact on the professionalism of 
educational technologists in improving the performance, strengthening the educational professional organizations. 
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However, the development of such standards should be continued for sustainable professional development. This will 
be beneficial for the development of Thailand's education, which affects the learner's ability to continue. 


Literature Review 


Educational Technology Standards in Other Countries 

For the development of international standards, the researcher has studied the standards of educational technology of 
AECT (2012), which has been defined five standards as follows: standard 1 content knowledge, standard 2 content 
pedagogy, standard 3 learning environments, standard 4 professional knowledge and skills, and standard 5 research. 
AECT is one of the world's leading educational technology providers, and Thailand is also one of the members of 
AECT. Therefore, a trend study of AECT technology standards is necessary. 

In addition, the researcher has studied the standards of the International Society for Technology in Education 
(ISET) is a not-for-profit organization focused on providing all educators with the ability to control technology to 
accelerate innovation in instruction and inspire learners to achieve high potential. It has developed standards for 
educational groups, including educators, coaches, teachers, and learners, details are as follows. 

ISTE Standards for educators (2017) is a standard that encourages learners to have greater potential for 
learning. It also promotes peer collaboration and interaction with other educators in finding new approaches. The 
development of various areas to support the learning of the students to the highest efficiency. It consists of 7 standards, 
the first standard is a developer of continuous learning, technology and creation and participation in local and global 
learning network, standard 2 is a leader in driving a visionary advancement in developing digital learning and learning 
that is equal to the student's success, standard 3 is a positive inspiration for learners in the digital world, standard 4 is 
a collaborative effort with others, using concepts, solving problems using technology, new digital resources to create 
a real-world learning experience, standard 5 is a truly learning activity designer by bringing tools, digital technology 
is used in many real-world environments, standard 6 is a facilitator of learning with the use of digital technology to 
support the development of learning outcomes of learners according to creative standards, and standard 7 is a 
knowledgeable and knowledgeable analyst who promotes alternative learning, designing and implementing multiple 
assessments to help learners achieve their learning goals. 

The part of the ISTE standard for teachers (2008) is defined in five aspects. 1) Teachers use content 
knowledge, teaching and learning by using technology to encourage students to learn to develop their creativity,2 ( 
teachers can design and develop learning experiences and assessments in the digital age,3 ( be a professional teacher 
who demonstrates competence, skill and expertise in innovation, 4) teachers promote and define responsibilities for 
digital citizenship, and 5) teachers participate in professional growth and leadership. The ISTE standards for teachers 
have been applied in the development of standards for students in order to improve their learning in various areas to 
increase their potential. 

Moreover, ISTE standard for student (2016) developed standards for learners by focusing on student-driven 
learning processes, consisting of seven standards 1) students use technology to demonstrate their potential to achieve 
their learning goals, 2) recognize liberty and rights, responsibility, opportunities of living, learning and working in 
the digital world, 3) value learning resources, especially digital tools, to be used in self-learning and others, 4) use a 
variety of technologies in the design process and solve problems to create new things, 5) develop and use problem- 
solving and understanding strategies with technology-assisted methods, 6) students can communicate creatively with 
digital tools and media, and 7) digital tools are used by learners to extend their perspectives and learning. It also works 
with others effectively in both local and global teams. 

However, the ISTE standards are the model used by states in the United States, which will be adapted to the 
context of their own state, for example the Washington State K-12 Educational Technology Standards (2018) have 
seven standards. Apart from that, Michigan adopted the ISTE Standards for Students as competencies for technology 
integration (MITECS) during the 2017-2018 school year, there are also Kentucky standards for teachers. Other 
countries have also adopted the ISTE standards, such as Pakistan. 


Educational Technology Standards in Thailand 


The development of educational technology standards in Thailand has been developed, but is still very limited and 
lacking continuity. Educational technologists’ roles are delineated in the Chapter IX of 1999 National Education Act 
1999 That the importance of educational technology has been determined in seven areas as follows: Educational 
technologists are professionals in the media by analyzing, designing, developing and providing media services in 
teaching and learning systems such as radio and television, text book, and printed media, as well as all types of 
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communication technology, while using research as a tool in the process, and practical professional development as 
well. 

The study or research on the development of educational technology standards or related is still low and lack 
of continuous development. However, there are also those who develop this. Thapanee (2003) has developed 
performance standard for educational technology in higher education institutions in Thailand consists of 14 standards 
and 84 indicators; 1) Instructional design and instructional development standard, 2) Instructional media design, 3) 
Educational technology training, 4)Research and development, 5) Internal and external assessment, 6) Educational 
technology diffusion, 7) Graphic media production, 8) Printed material production, 9) Video/television production, 
10) Audio material/radio production, 11) Computer media production, 12) Education technology consultation service, 
13) Instructional media and audio visual aids service, and 14) Planning and management. 

In addition, Wasan et al. (2006) developed the national standard for educational technology for the institute 
of graduate study. It consists of 9 standards, 34 indicators as follows: 1) the institutional leadership in educational 
technology, 2) the curricular structure of teacher education program, (3) the technology infrastructure of educational 
technology, 4) the technology innovations fostering the instruction in teacher education institution, 5) the personnel 
in educational technology, 6) the faculty members’ performance in educational technology, 7) the instructional models 
fostering the pre-service teacher students’ uses of educational technology, 8) the professional experiences, and 9) the 
pre-service teacher students’ performance in educational technology. 

From literary review by analyzing, synthesizing the research papers related to the standard, it is shown that 
Educational technology standards need to be developed urgently and should be continually developed to create 
professional standards for technologists to be strong and up-to-date with the rapidly changing digital technology world. 
For the benefit of education systems that affect learners’ learning and instructor effectiveness to maximum potential. 


Research Objectives 
The study is aimed to research standards and competency for Thai instructional technologists. 
Research Methods 


Researchers have developed a study on the development of educational technology standards of each organization 
both in Thailand and abroad as follows. 

1. Study and analysis of educational technology standard documents for educational technologists as well as 
teachers and educational personnel of foreign agencies. 

2. Develop a draft standard for educational technology based on the review of research and educational technology 
standards from organizations in other countries. 

3. In-depth interviews with experts in the field of educational technology were conducted to gather information, 
opinions and expert advice on the drafting of educational technology standards. 


Research Results 


According to studies, it has been found that researchers can synthesize draft standards of educational technology has 
six standards, as follow. 

Standard | Profound knowledge in technology and its adaptation to education has three indicators include: 
1.1) profound knowledge of both theoretical and practical education technologies, 1.2) profound knowledge about 
education, 1.3) knowledge of new technologies. 

Standard 2 System design has 2 indicators include, 2.1) design of teaching and learning system, 2.2) design 
of media system. 

Standard 3 Professional development has three indicators: 3.1) expertise in media production skills, 3.2) 
leadership in new technologies applied in teaching, 3.3) be a consultant to develop learning by using advanced 
technology. 

Standard 4 The learning environment has two indicators: 4.1) the development of a learning environment 
that uses technology in various ways, 4.2) the organization of a diverse learning environment based on real conditions. 

Standard 5 Research and ethics have two indicators: 5.1) Applied research in education and educational 
technology, 5.2) Research in development of media, 5.3) Ethics of Educational Technology and rese. 

Standard 6 Localized and globalization has two indicators: 6.1) the application of knowledge in the field of 
technology and the use of local networks, 6.2) the creation of a broader network to the global network. 
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In terms of in-depth interviews of experts with interesting issues, as detailed below. 

Standard 1 Profound knowledge in technology and its adaptation to education. "Standards 1 should identify 
indicators of educational attainment in relation to teacher professional standards and student performance in Thailand 
as teachers and learners are key target groups for educational technology. Therefore, standards of educational 
technology need to be consistent with the standards and competencies of teachers and learners." (Experts 1, 4 and 5) 

Standard 2 Design System "should include theories of learning, psychological theory, perception theory, 
media characteristics in design." (Experts 3, 4 and 7) 

Standard 3 Professional development “Identify skills, expertise in thinking, creativity, use of all learning 
materials, and technology in advanced platforms.” (Experts 2, 5 and 6) 

Standard 4 Learning environment “The ability to utilize the atmosphere and suitability of the classroom and 
other learning facilities should be discussed.” (Experts 3, 4 and 7) 

Standard 5 Research and ethics "Standard 2 should focus on innovative research to be in line with the 
Thailand 4.0 policy and should include research ethics, progressive thinking, application and contemporary 
thinking.” (Experts 1, and 3) 

Standard 6 Localized and globalization “Identify the actual implementation. In accordance with the policy 
of Thailand 4.0, but consistent with the culture of culture in the context of Thai society.” 

(Experts 2, and 5) 

In addition, experts have proposed that additional standards or clear indications be added. 

1) Leadership standards should be added as most educational technologists play the role of a leader that is 
not as clear as it should be. Therefore, it should be defined as one standard that will improve the performance of 
educational technologists to a higher level. (Expert 2, 3 and 6) 

2) The standard of the role counselor or facilitator of educational technology should be identified. (1, 3, 4 
and 7) 


Discussion and Conclusion 


The standards developed by the researcher consist of six standards, 1, 3,4 and 5, consistent with the AECT (2012) 
standards, but differ in their metrics and focus. Standard 1 Content Knowledge has a different focus: Standard: 1, 
which develops profound knowledge in technology and its adaptation to education, focuses more on the knowledge 
of the educational system that involves teachers. Next, standard: 3 professional development, for example, defines an 
indicator that identifies media production skills, being a consultant, and a technology leader, which are not the same 
thing. Standard 4: the learning environment is focused on the environment where technology is used to learn and the 
learning environment is realistic. And standard 5: research and ethics, the emphasis is on the name of the standard, 
the code of ethics, which shows the ethics is important and the research has two indicators, namely, applied research 
and development research. 

While standard2: system design, conforming to standard 2 of ISTE For Teachers (2008) Some of the design, 
development, evaluation, and learning outcomes of learners. It does not mention the system. The standard 6: localized 
and globalization complies with ISTE standards for educators (2017) in Standard: 1 and Standard: 7 for students. 
However, based on expert interviews, some ideas need to be added to the standards that define the role of educational 
technology more clearly: the standards of leadership and the standards of counsel and facilitation. In accordance with 
the ISTE Standards for Teachers (2008) and ISTE for Educators (2017). 


Conclusion 


The standard of educational technology in Thailand consists of 6 standards; 1) profound knowledge in 
technology and its adaptation to education, 2) system design, 3) professional development, 4) learning environment, 
5) research and ethics, and 6) localized and globalization Based on interviews with experts, it was found that the 
experts gave more suggestions on adjusting standards for greater coverage. The researcher has to take this effect to 
further study and make appropriate adjustments. The results of this study are consistent with the recommendations of 
the panel of experts to ensure that the educational technology standards are fulfilled before proceeding with the 
research. 
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Abstract 


An online monitoring and evaluation system was developed to monitor and evaluate students' reading activities. This 
system was designed as an online learning environment (OLE). As the students display more community 
characteristics on OLE, the environment is getting stronger accordingly. At this point, it is wondered how student 
behaviours explaining the characteristics of being a community and how these behaviours affect other behaviours. In 
this research, an answer to the following research question was sought, "Is there a relationship between the number 
of log in OLE and the number of completion of tasks? What is meant with the number of log in the environment is 
the number of log in the online monitoring and evaluation system here, and what is meant with the number of task 
completion is the number of books read. Online monitoring and evaluation system was used during a school term at 
a Turkish secondary school. Participants of this research were 412 secondary school students. In this quantitative 
research, log files were used to collect data. Online secondary school. Participants were 412 secondary school 
students in the quantitative research. At the end of the study, it was found that both low and high-level readers were 
able to log in the system at a low level. This can be interpreted as that high-level readers focus on completing the 
tasks they are directly involved in, and therefore, they contribute to the system at a low level. However, it was 
observed that those who logged in the system at a high level and read many books (17.5%) were found to logged in 
the system significantly more than those who logged in the system at a high level and read fewer books (8.7%). 


Keywords: Online learning environments, student behaviour, student participation behaviour, task completion 
behaviour 


Introduction 


The high rate of the reading book in a society is a sign that there are many people who have critical 
thinking skills in that society. Reading book is beneficial for individuals and thus the societies. Because reading is 
the basis of mental development, which is an indicator of the level of development of society. In order to help 
students gain the habit of reading books, students need to meet the books appropriate to their level and interests, and 
their reading processes should be also followed (Bull and McKenna, 2004; Williams, Bialac and Liu, 2006). After 
reading a book, students' readings should be evaluated (Hickman, Bielema and Gunderson, 2005). An online 
monitoring and evaluation system was developed to monitor and evaluate students! reading activities. This system 
was designed as an online learning environment (OLE). Different roles and behaviours have been suggested in the 
literature for strengthening the OLEs. In these strengthening studies, the main focus is on community characteristics 
of these online learning environments. The general opinion in the literature is that in the cases where OLEs happen 
to become strong communities, the success in the OLEs is high (see Yang, Wang, Shen & Han, 2007; Yeh, 2010). 
Different researches have examined what type of behaviours are necessary for these environments to make the 
environment efficient and to make the environment a community. The analysis of student behaviours in OLE is 
necessary to understand what the outcome of the behaviours is and to determine which behaviours are supported for 
learning. 


In the literature, there are studies investigating which student behaviours are developed in OLEs (eg 


Kneser, Pilkington & Tresasure-Jones, 2001), which roles form which behaviours (eg Yeh, 2010) to understand the 
outcomes and functions of OLE on students’ developed behaviours (eg De Wever, Schellens, Valcke and Van Keer, 
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2006). It is understood from the results of the studies in the literature that student behaviours can be influenced by 
the environment and roles, and behaviours can affect OLE and even student behaviours can be used as an indicator 
to understand the efficiency of OLE. In this framework, it is worthwhile to understand and interpret student 
behaviour in OLE. 


It is understood from the results of the studies in the literature that student behaviours can be influenced by 
the environment and roles, that behaviours can affect OLE and even student behaviours can be used as an indicator 
to understand the efficiency of OLE. In this framework, it is worthwhile to understand and interpret student 
behaviours in OLE. In some studies aiming to examine the nature of the environment through the discussion records 
in the OLE, the frequency of participation of the students in the environment, the forms of participation (with or 
without task completion, active / passive etc.) and the content they shared in the environment were considered as the 
behaviours of students in the OLE (Blignuat and Troppil, 2003). It is also stated that, through these behaviours, 
predictions can be made regarding students’ gains (De Wever, Schellens, Valcke and Van Keer, 2006). 

In some of the studies evaluating student behaviours in the cognitive category, asking questions, participating in 
discussions, explaining, criticizing and developing suggestions were examined as student behaviours. Some studies 
were categorized within the scope of student behaviours. Some examined all kinds of participation in the field as 
academic behaviours and the others examined them as non-academic behaviours. Beuchot and Bullen (2005) 
examined student behaviours through social behaviours. Yeh (2010) states that the roles performed shape the 
behaviours of the individual, and he classified student behaviours by analysing them. The classification of the 
behaviours are as follows: (1) high cooperation and high participation (Active collaboration), (2) high cooperation 
and low participation (Passive collaboration), (3) low collaboration and high participation (Individualized 
participation) and (4) low cooperation and low participation (Indifference) 


Research Question 


In the study, the answer to the following question was sought “Is there a relationship between the number 
of participants in the OLE and the number of task completion?” Here, what is meant with the number of log in the 
environment is the number of log in the online monitoring and evaluation system, and what is meant with the 
number of task completion is the number of books read. 


Method 
Research Context 


Researchers conducted the study at the schools in a province located in the western part of Turkey through 
an Online monitoring and evaluation system aiming to improve students' reading habits. This system was designed 
as an OLE. The online monitoring and evaluation system for book reading activities which was used as an OLE was 
a system funded by TUBITAK (The Scientific and Technological Research Council of Turkey) and developed by 
the Canakkale Onsekiz Mart University. 


During one semester, the OLE was used in two schools. The use OLE measures students! reading habits 
through a scale at their first log in the system. The system identifies the students’ areas of interest in reading books 
in their first entry and provides them with the opportunity to change their preferred fields of interest anytime along 
the process. OLE starts to recommend the most appropriate books to the students choosing among the ones available 
in the system considering the parameters defined through the scale and questionnaire given at the very beginning. 
The book recommendation system included in the OLE aims to assist students in the book selection process and to 
prioritize the books that the student may like in accordance with their habits of reading books. Students can choose a 
book that is recommended to them, or they can start to read a book which they themselves prefer. 


Students obtain and read the book they want to study. There is no book reading session in OLE. Once 
students inform the system that they have decided to read a book, the OLE continues to guide students along the 
reading process. In OLE, students are informed about the number of people reading the chosen book, the number of 
people who like it, the number of people advising and the time recommended to read the book. The students who 
complete the book reading process states that in the system to complete the reading process. Upon this, the OLE 
asks students if they are ready to make the evaluation process in order to determine how well the chosen book was 
comprehended. Students can have a multiple-choice test in OLE. This test consists of 10 multiple choice questions 
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randomly chosen from the question pool to measure how well the read book is comprehended. The student's reading 
comprehension score is sent to the screen of teachers, students and their parents. After this stage, students are asked 
whether they want to write any comment about the read book, to recommend it to friends and whether they want to 
rate the book with star/stars. During one semester, students continue this process through the OLE. 


Participants 


The participants of the study were 412 secondary school students (188 male and 224 female). 211 of these 
students were Sth grade (51%), 139 of them were 6th grade (33%), 42 of them were 7th (10%) and 20 of them were 
8th grade (6%) students. 


Data Collection and Analysis 


The data collection tool of the study were the records of the OLE system. The OLE system counts how 
many books students have read. The number of books that students have read is considered as “completed task” 
behaviours. The number of students’ log in the system is considered as “participation” behaviours. The system also 
keeps records regarding this data. The relationship between the number of students’ log in and the number of task 
completion was found out by chi-square analysis. 


Findings 


The chi-square analysis was used to examine whether there was a relationship between participation in 
OLE environment and number of task completion. The results are given in Table 1. 


Table 1. 
The relationship between the number of task completion and the number of log in the system 


The number of Log in the System 
Low level (0-14) High level (15 and 


above) Total 
Number of n 303 29 332 
books read POMC N NEE): ay 91.3% 8.7% 100% 
(Number of task High level (3 n 66 14 80 
completion) and above) % 82.5% 17.5% 100% 
Total n 369 43 412 
% 89.6% 10.4% 100% 


(X2=5.30, df=1, p=.021) 


Low-level readers logged in the system at a low level (91.3%). High-level book readers logged in the 
system at a low level (82.5%). In other words, both high-level readers and low-level readers logged in the system at 
a low level. However, when we looked at the high level of log in the system, it was seen that those who read a lot 
(17.5%) were found to have logged in the system significantly more than those reading book very little (8.7%) (X2 
(df= 1) = 5.30, p <.05). 


Conclusion and Suggestions 


In this study, the relationship between students' logged in OLE environment and task completion 
behaviours were investigated. In the study, it was found out that both low book readers (91.3%) and high book 
readers (82.5%) logged in the system at a low level. This can be interpreted as that high-level book readers focus on 
completing the tasks they perform directly and therefore contribute to the system at a low level. It is possible to 
suggest that low-level book readers log in the system less as they have less number of task completion. In the study, 
it was also observed that those who logged in the system at a high level and read a lot of books (17.5%) logged in 
the system significantly more than those who had a high level of log in the system and read fewer books (8.7%). It 
can be interpreted from that the log in the system has an effect on the reading rates. Further studies are needed to 
investigate the relationship between different student behaviours. Moreover, under the light of the findings of this 
study, the researchers can examine the effects of the roles designed for instructional purposes and how they can 
influence students’ behaviours. 


308 


References 


Beuchot, A., & Bullen, M. (2005). Interaction and interpersonality in online discussion forums. Distance Education, 
26(1), 67-87. 

Blignuat, S., & Troppil, S. R. (2003). Developing a taxonomy of faculty participation in asynchronous learning 
environments: An exploratory investigation. Computers & Education, 41, 149-172. 

Bull, J., & McKenna, C. (2004). Blueprint for computer-assisted assessment. London: Taylor & Francis. 

De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of 
online asynchronous discussion groups: A review. Computers & Education, 46(1), 6-28. 

Hickman, C. J., Bielema, C., & Gunderson, M. (2005). Challenges in the Design, Development, and Delivery of 
Online Assessment and Evaluation. S.L. Howell & M. Hricko (Eds.). Online assessment and measurement: 
Foundations and challenges (pp. 132-164). IGI Global. 

Kneser, C., Pilkington, R., & Tresasure-Jones, T. (2001). The tutor's role: An investigation of the power of exchange 
structure analysis to identify different roles in CMC seminars. /nternational Journal of Artificial 
Intelligence in Education, 12, 63-84. 

Williams, G. C., Bialac, R., & Liu, Y. (2006). Using online self-assessment in introductory programming 
classes. Journal of Computing Sciences in Colleges, 22(2), 115-122. 

Yang, F., Wang, M., Shen, R., & Han, P. (2007). Community-organizing agent: An artificial intelligent system for 
building learning communities among large number of learners. Computers & Education, 49, 131-147. 

Yeh, Y. (2010). Analyzing online behaviours, roles, and learning communities via online discussions. Educational 
Technology & Society, 13(1), 140-151. 


309 


Reflection Wrapper Activities to Promote Pre-Service 
Teachers’ Metacognitive Strategies 


Amani Bin Jwair 
Department of Curriculum & Instruction 
Southern Illinois University 
Email: amani2007@siu.edu 


Abstract 


This paper investigates the impact of reflection wrapper activities to improve pre-service teachers’ 
metacognitive strategies in the higher education classroom. It reviews relevant literature describing what reflection 
wrappers are, how to use these activities, and their significance to enhance pre-service teachers’ metacognitive 
strategies, such as planning, monitoring, and evaluating. 


Introduction 


Prior research suggests that when students are shown how to self-regulate their study behavior and 
knowledge, it could have long-lasting positive effects on their learning (Gezer-Templeton, Mayhew, Korte, & 
Schmidt, 2017; Lovett, 2013). These skills can be developed gradually during class time by teaching alternative 
study strategies or outside the classroom by giving students assignments to practice these skills. For example, 
metacognitive strategies that teachers want to promote in education can be embedded easily by employing a tool 
known as an exam wrapper. One of the desired outcomes of exam wrappers is enabling students to self-evaluate and 
make plans for improved future study strategies. Lovett (2008) developed the exam wrapper concept, while exam 
wrappers were first introduced as a tool in a posttest analysis by Achacoso (2004). The reflective post-exam 
questionnaire has since formed the backbone of the exam wrappers that are used today. 


Reflection Wrappers 


The reflection wrapper includes diagnostic questions that facilitate student self-assessment and 
understanding of lessons, help with exam preparation, and promote metacognitive skills. These tools have gained in 
popularity by prompting students to reflect on three major components of learning: exam preparation (thinking about 
study skills used), types of errors made in study or on exams, and adjustments for future learning (modifications to 
study habits to better prepare for the next class or exam). Reflective wrappers include exam wrappers and reflective 
thinking activities. 


Exam Wrappers 


Exam wrappers are short questionnaires administered around the end of a lecture, assignment, or 
assessment to train the student to think through the steps of metacognition, i.e., planning, monitoring, adapting, and 
evaluating (Root Kustritz & Clarkson, 2017). Lovett (2013) described exam wrappers (or cognitive wrappers) as 
concise, straight-forward activities that urge students to assess their performance (and the instructor’s feedback) on 
an exam with directions for the future in mind. Gezer-Templeton et al. (2017) described them as brief, reflective 
writing activities that ask students to evaluate their applied study approaches to their performance on an exam while 
concentrating on modifying future learning practices. For example, students can fill out exam wrappers after they 
take a performance exam. Such wrappers include short questions asking about their exam preparation, what the most 
difficult questions were, and what they might do differently before the next performance exam. Unfortunately, there 
are few published studies on the use of exam wrappers as a strategy in a blended learning environment (e.g., a 
flipped classroom) to improve metacognition. 
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Reflective Thinking Activities 


In addition to exam wrappers, reflective wrappers also include reflective thinking activities that encourage 
students to engage in active learning through problem-based tasks to develop higher-order thinking skills while 
thinking about their thinking. For instance, the instructional design of the flipped classroom tends to have activities, 
such as pre-class asynchronous metacognitive activities, and to record lectures out of class time. This affords more 
opportunities to engage students with questions about the content and discuss difficult concepts or topics in active 
learning. Thus, students are empowered to explore and solve problems in groups to achieve their learning goals 
before class. Students are expected to demonstrate an understanding of the task and topic by problem solving on a 
discussion board before class to build their basic knowledge of the content and discuss complex tasks with friends or 
the instructor outside the classroom (such as online) through a pre-class asynchronous metacognitive activity or 
inside the classroom (via face-to-face learning). 


Significance of Wrappers 


Exam wrappers include three basic types of questions: 
1. “How did you prepare for the exam? 
2. What kinds of errors did you make on the exam? 
3. What could you do differently next time?” (Lovett, 2013, p. 23) 

Responding to the exam wrapper helps students develop self-regulated learning (SRL) skills (Gezer- 
Templeton et al., 2017; Lovett, 2013) and metacognitive strategies (Lovett, 2013). SRL involves learning certain 
skills, such as self-assessment, goal setting, goal implementation, and monitoring progress (Zimmerman, 2002). 
Wrappers do not take up much class time, require minimal time from the student, are easy to adapt across different 
courses, and address several components of metacognition (e.g., performance evaluation, assessment of strengths 
and weaknesses, behavioral adjustment, and strategy identification). 

The exam wrapper helps students learn more from test results and improve their performance on future 
tests (Lovette, 2013). It has thus come to be seen as an effective and valued post-exam reflection tool for promoting 
self-study habits (Gezer-Templeton et al., 2017). It also assists teachers in suggesting learning strategies and 
providing encouragement (Butzler, 2016). Furthermore, exam wrappers have allowed students to reach their own 
conclusions about the learning process (Lovett, 2013), and they are an effective tool for helping learners use their 
developing metacognitive strategies (Butzler, 2016; Lovett, 2013; Soicher & Gurung, 2017). 

Based on sound pedagogical and theoretical principles, the reflective wrapper aims to foster metacognitive 
skills by training students to recognize what information they accurately understand, so they can devote more effort 
and time to information they do not understand (Root Kustritz & Clarkson, 2017; Lovett, 2013). Reflective wrappers 
could therefore help independent lifelong learners continuously assess the outcomes of their actions to build new 
knowledge. 

The literature has reached a common understanding of what wrapper activities are. However, there is still a 
pressing need to understand how and when instructors can implement reflective wrappers and connect these 
activities to an environment with a mechanism for training metacognitive skills while meeting the criteria for 
applying exam wrappers (e.g., short questions, repeated use, use immediately after students receive their graded 
activity or performance exam). It is also important to connect reflective wrappers to evidence of advantages to 
different aspects of student study habits. 


Implementing Exam Wrappers 


It is easy to implement exam wrappers due to several factors (Butzler, 2016; Lovett, 2013). First, they are 
short (1-2 pages), often containing only short-answer questions. Second, they do not require much student time to 
complete or teacher time to assess. Third, teachers can easily adapt the exam wrapper for different majors and any 
learning task. For example, Lovett (2008) asserted that exam wrappers could be used for assessing lectures, 
homework, quizzes, and discussion. Fourth, exam wrappers are repeatable tools for subsequent quizzes, with some 
changes to avoid repetition. Fifth, this reflection helps students evaluate their performance, assess their weaknesses 
and strengths, identify the best strategies that work for them, and make appropriate adjustments. The metacognitive 
strategies students practice by using exam wrappers in a course provide substantial benefits, and teachers can 
encourage students to apply these strategies to other tasks and classes (Lovett, 2013). In higher education, beyond 
checking their grades, students typically choose to put the test out of their mind and proceed on to the next task. 
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Nonetheless, it is imperative for educators to carefully explain the reasons behind students’ performance on a given 
test (Lovett, 2013). 

Lovett (2013) explained the primary method of using exam wrappers. Students prepare and ask about the 
first exam using their regular study strategies. No intervention is required before the exam. The teacher has students 
complete exam wrappers after their graded exam is returned to them. The teacher collects the exam wrappers, not to 
grade activities but rather so they can be returned to the learners before their next exam. The teacher will also want 
to review students’ responses to gain insight into their learning. This practice leads to a better understanding of how 
the students evaluate their own strengths and weaknesses. The students must begin studying and preparing for the 
next exam once they are done reviewing any recommendations on the first exam wrappers to include their own 
suggestions. 

A review of the literature uncovered certain points of controversy. For example, some studies found that 
students did not appear to change their behavior after using exam wrappers, while other studies had positive results. 
The goal of implementing exam wrappers in a universal design for instruction is to train students in metacognitive 
skills and meet the criteria of successful exam wrapper use. This in turn could reveal how to imbed metacognition 
within coursework, providing insight into future directions for pedagogical practices with wrappers in schools and 
universities. This activity allows students to practice metacognitive skills and see an immediate change in their 
academic performance. 

Lovett (2013) evaluated the use of exam wrappers in four courses across multiple disciplines (calculus, 
chemistry, introductory biology, and physics) to enhance metacognitive strategies during an academic semester. The 
findings demonstrated no change for students enrolled in only one course, and there was no control group. Higher 
gains were seen when learners used the exam wrapper across the four courses, but the researcher did not address 
whether using them in one course was more beneficial than not using them at all. She recommended that 
metacognitive skills be taught like any other skills and that they benefited from feedback and practice. Showing 
even less positive results, Butzler (2016) investigated the effects of implementing SRL tools (e.g., exam wrappers 
and notetaking) on chemistry students’ achievement. The findings of a paired-samples ¢-test showed no significant 
difference with the addition of SRL tools. 

Root Kustritz and Clarkson (2017) investigated the use of exam wrappers to help students in a first-year 
veterinary anatomy course self-assess their quiz preparation and employed a control group. Students in the 
experimental group filled out an exam wrapper after they completed the second and third out of four quizzes. The 
exam wrapper contained questions about quiz preparation, where they felt they had the most trouble with the quiz, 
and what they might do differently before the next one. The findings demonstrated no high percentage change in 
scores from the second to third or third to fourth lecture or quiz; however, the final exam scores improved for the 
experimental group. The researchers attributed students not changing their behavior from one quiz to another to a 
probable lack of formal training in metacognition. As a result, students did not understand the value of completing 
the exam wrappers or the potential advantages of utilizing their reflections. The researchers mentioned that a future 
study could describe the results when learning objectives particular to metacognition are involved in coursework in 
the veterinary curriculum. 

In contrast, Achacoso (2004) found positive effects from using exam wrappers. Students increased their 
metacognitive strategies after using them, which was confirmed by their ability to monitor and adjust study 
strategies and an overall increase in their mean exam scores. Similarly, Thompson (2012) investigated using exam 
wrappers to enhance metacognitive self-monitoring practices among college students in a Spanish course and 
compared the results to a control group. The researcher used comprehensive testing to promote greater use of self- 
monitoring practices with modest results. The exam wrapper group did not show more gains in self-monitoring 
strategies—as measured by the metacognitive self-regulation scale of the motivated strategies for learning 
questionnaire—compared to the group without exam wrappers. However, the analysis showed that students in the 
control group were more proficient in Spanish at the beginning of the study, and the self-monitoring practices 
changed in the experimental group when students completed posttest reflection exercises and talked explicitly about 
study strategies in class. In other words, the mean metacognitive self-regulation score increased for students who 
had completed exam wrappers. 

Gezer-Templeton et al. (2017) investigated students’ metacognitive strategies, assessed the correlation 
between study behavior and student performance, and evaluated students’ perceptions of exam wrappers. Exam 
wrapper activities were given as an extra credit assignment after the first three exams in a food science and nutrition 
course. The researchers analyzed student responses and exam performance, and the findings showed that most 
students with poor exam performance overestimated their exam scores, which meant students’ self-assessment skills 
could be improved. However, the results also demonstrated the ability to make and implement goals to develop 
study strategies during the semester. The researchers observed a modest correlation between improving exam 
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performance and using study strategies, especially for students with a B average on exams, indicating that students 
within the middle of the grade distribution may benefit most from exam wrappers. Most students believed that exam 
wrappers helped them improve their study habits and exam scores, and they planned to use the exam wrapper 
process in future classes. 

One of the most important requirements for creating a suitable environment for improving metacognition is 
that learners must prepare for scheduled classes by reviewing the class content online. This preparation enables 
learners to interact productively with their peers and teachers by actively participating in discussions on in-depth 
questions during class. Thus, learners may need to be more motivated to study the class content ahead of time. This 
motivation can be achieved through the integration of SRL in a flipped classroom. 

Cakiroglu and Oztiirk (2017) investigated problem-based learning in a self-regulated flipped-learning 
(SRFL) approach to develop the SRL skills of 30 male and female Mechatronic undergraduate students. A 
qualitative design was used to collect the data through discussion-board messages, an observation form, and 
interviews with select participants. The results showed that students’ skills improved but were still moderate. 
Learning outside the classroom helped them develop environment structuring, goal setting, and planning skills at a 
high level. The study recommended examining the relationship between the SRFL approach and problem-based 
learning in different disciplines. Future research could employ an SRFL approach by integrating SRL activities with 
the flipped-learning approach, and a mixed-methods design could be used for different SRL strategies. Future 
studies could also use pre-class asynchronous metacognitive activities with the self-reflection metacognitive 
instrument and exam wrappers. 


Conclusion 


This literature review addressed exam wrapper issues from different perspectives, providing insight into 
future avenues of research. However, the study could not answer all concerns about how to implement reflective 
wrappers to improve metacognition. More research is thus needed to answer this question and determine whether the 
flipped classroom is a better learning environment to implement and promote metacognitive skills and evaluation 
practices by having students think about their own thinking. 
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Abstract: 


Mission HydroSci (MHS) teaches water systems and scientific argumentation toward meeting Next 
Generation Science Standards. MHS is a game-based 3D virtual environment for enacting transformational 
role-playing, wherein students must learn new knowledge and competencies in order to achieve the game 
missions. MHS was developed for middle school science as a replacement unit of about 6 to 8 hours using 
analytics and a teacher dashboard to support teachers supporting students. 


Introduction 


Mission HydroSci (MHS) is a game-based 3D virtual environment for teaching and learning key 
concepts and knowledge of water systems and building competencies in scientific argumentation in ways 
that support meeting Next Generation Science Standards (NGSS Lead States, 2013). Meeting these new 
science education goals for middle school students requires rich learning contexts for exploring substantive 
science ideas through engagement in scientific practices. Our goal is to meet these educational goals for all 
learners by using online learning to serve those in distance education and rural communities as well as 
more traditional and well-resourced classrooms, and to use gaming to engage and support students who 
typically do not see themselves as successful science learners. 


MHS is a research and development project funded by grants from the U.S. Department of 
Education. The grant support has enabled a team of researchers, science educators, learning and game 
designers, creative arts professionals and software developers to build MHS through an iterative process 
(Laffey, et al. 2105; Laffey, et al. 2016). The process started with envisioning a powerful fit between 
emerging technological capabilities and the requirements of teaching and learning to NGSS. Next came 
conceptualizing a complete system to engage and teach students a robust curriculum and then building and 
testing each component of the system. The building and testing process has included prototyping, creating 
design documents such as requirements specifications and storyboards, building initial versions, which can 
be taken to usability testing and then refined. Once we felt a substantial portion of MHS was playable and 
met our requirements specifications for teaching and learning activities to implement the curriculum as a 
game, we conducted usage testing in live classrooms. The usage testing taught us about the practical 
challenges of using a game in classrooms as well as identifying many areas of game play that needed 
improvement. Our intention and obligation to the funding sources is to undertake a field test using a 
randomized control trial (RCT) to rigorously evaluate the impact of MHS game play. The RCT will be 
undertaken in Winter, 2019. However, to test the feasibility of conducting a large field test in classrooms 
we undertook a preliminary field test in the Spring of 2018. This report and showcase describes MHS and 
presents some insights about the use of MHS in classrooms from interviews with 12 teachers who 
participated in the 2018 feasibility testing. 


Rationale for MHS 


While MHS is a research project with goals of understanding and testing the potential impacts of a 
gameplay approach on teaching and learning and of developing game play strategies that map to teaching 
and learning approaches, it is also a product development project aspiring to build a game that middle 
school science teachers will use to meet important learning objectives in their science curriculum in ways 
that align with the NGSS. MHS targets general and earth science courses by meeting learning objectives 
for understanding water systems and building competencies in scientific argumentation. The MHS game 
provides an active learning environment for meeting these learning objectives by engaging students in a 
narrative about needing to investigate water resources and use scientific argumentation to complete 
missions critical to the survival and accomplishments of the members of their scientific enterprise. The 
enterprise is set on an earth-like planet in the future as the science cadets (our player and a set of non-player 
characters who serve as guides, partners and sometimes antagonists) explore mysteries and prepare for 
survival on the planet. Along with the narrative game play MHS includes learning progressions for water 
systems science and scientific argumentation, a visually exciting environment, substantial interaction and 
feedback, and applies transformational role playing as an approach to integrate learning with game play. 

The theory of transformational play (Barab, Gresfali & Ingram-Goble, 2010) shows how specific 
design strategies can optimize the potential of games and gameplay to lead to desired learning outcomes. 
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The design strategies to enact transformational play include the student taking on the role of a protagonist 
who must use subject matter knowledge to make decisions and take action during play, and having these 
actions and decisions transform the problem-based situation. In turn the student’s understanding of the 
subject matter is transformed and so is the student’s identity. Virtualization and role playing experiences in 
MHS are intended to make realistic actions possible and bring the consequences of actions to dynamically 
impact the world and the learner. 

Building competence with scientific practices, such as scientific argumentation, requires learning a 
progression of competencies, thus multiple practice settings and iteration with feedback must be provided. 
The game experience helps sustain the student through the many activities as well as makes it OK and 
natural to fail and try again. MHS is a first person narrative adventure with a sustained learning experience 
of 6 to 8 hours of instructional time for gameplay and | to 2 hours of supplementary classroom or 
discussion board activity for the teacher to clarify, supplement, and extend the learning from the gameplay. 


Game Play 
Unit 1 


Unit 1 introduces students to (1) gameplay including game controls, characters and narrative, (2) 
scientific argumentation as a process of using evidence to judge between competing claims and (3) the 
argumentation engine that will be used to conduct arguments during game play. The design task of unit | is 
to help the player get off to a good start, but also to set the stage for engaging at one’s own pace, so as to 
learn how to succeed in the game and not just to move through the game. The tutorial nature of some of the 
starting tasks are counter balanced by interesting and fun visuals, learning about interesting NPC 
characters, and an exciting start to the game. 

The unit starts with the player awakening on a space station orbiting an alien but earth-like planet. 
The player is introduced to ARF who will be an assistant for the player’s exploration and activities. ARF is 
presented as a buddy, given an avatar of a dog, and a high form of artificial intelligence to assist the player. 
The player also meets Dr. Toppo who is the mission leader and sets up challenges for the player throughout 
the game. After learning some basics of how to navigate and play MHS as well as being introduced to other 
NPC’s and tools to be used throughout the game, the space-craft if rocked with an explosion and our player 
must escape the station and fly to the alien planet. 

i U1- Time to Argue! 2 —<«£ 
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Figure 1: Meeting Dr. Toppo and ARF on the spacecraft and the explosion. 


Unit 2 


Unit 2 teaches players about topography and using a topographic map as well as understanding 
watershed and how the relative size of the watershed is related to the amount of water flowing through it. 
After crash landing on the alien planet our player practices some of the skills they learned in unit 1 while 
collecting scrap to repair a broken hoverboard and tracking down the communication equipment which 
they will need to move rapidly on the large terrain and find the rest of their team. In order to find the team 
they must use a topographic map and respond to clues and feedback framed using topographic terminology. 
Once they have located the team, our player is assigned the task of finding which watershed is larger and 
thus best for setting up the base camp for the expedition. After traveling to key waterfalls and gathering 
evidence our player engages in an argument where she must use evidence to support the claim of which 
watershed is largest. 
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Unit 3 


Unit 3 teaches about surface water with the learning objectives of having the player be able to 
predict the spread of a dissolved material through a watershed and identify the direction of water flow 
based on a map of a watershed. This is enacted narratively through one of the NPCs, Sam, who needs to set 
up her camp but her supplies are scattered all over the terrain. Our player must find the supplies and figure 
out which waterways to use in order to float them back to Sam’s camp. Upon completing the task, our 
player returns to the base only to find that Sam’s base is polluted and she asks our player to solve what’s 
causing the pollution by tracing the source of the pollution. The player does this by throwing sensors into 
the nearby river and eventually will find wreckage from the space station explosion. Our player works their 
way up the river and then must create a good argument using reasoning to connect evidence with a claim in 
order to get the pollutant removed. Next, our player discovers that the aliens had left irrigation devices to 
support growing food in gardens, which will be extremely important for the success of the expedition. 
Unfortunately, the pumps are old and need to be replaced. By succeeding at solving the first of the 
dungeon-like puzzles the player unlocks new pumps and then must use their knowledge of surface water 
flow to pick which irrigation systems to restart. 


Sam's base is polluted! Sam's base is polluted! 
Return to Sam ‘Trace the poftutant 


Figure 3: Sam telling you that the crate was tossed in the wrong river and tossing sensor in river to collect 
evidence for finding the pollution. 


Unit 4 


Unit 4 teaches about groundwater with objectives of understanding water tables, predicting rates 
of infiltration based on permeability of the soil type, and explaining the movement of water from the 
surface to the ground system. Our player is sent off to explore recently discovered ancient alien ruins and 
finds a drill that the aliens apparently used to access underground water. The drill isn’t currently 
operational, so the player must solve the second puzzle in order to restart the drill and access the water 
table. This provides a source of water for the expedition but also has the downside of flooding the new 
bunker facility that Riley, another NPC, had just built. Our player must then explore the bunker to identify 
if the flooding was caused by drilling or by Riley having built the bunker in the wrong place. Our player 
must go through the bunker collecting soil samples in order to build a model of how the underground water 


317 


will flow and the permeability of the different soil types. By doing this the players collects evidence for 
which claim is correct and subsequently answers if it was player or Riley’s fault for the flooding. Our 
player must construct 2 complete arguments with claims, reasoning and evidence to sort out fault. Next our 
player must engage in an engineering task to identify which fountain to flood in order to send water to the 
base camp. 


Alien Village 
— 


Figure 4: Finding an alien village in the desert and using their drill room to access underground water. 
Unit 5 


Unit 5 teaches about the movement of water through a cycle focusing on state changes that occur 
in atmospheric water.. The learning objective is for students to understand the energy required for 
atmospheric phase changes. The player arrives on an tropical island at the request of one of other other 
cadets, this time Bill. Bill wants to show the player a process that he just discovered, but first the player 
must solve another of our puzzles. This puzzle requires students to transport seaside water by changing its 
phases from solid, gaseous and back. By doing this, the player also gains a better understanding the 
relationship of humidity and temperature. After successful completion the player follows Bill to learn about 
his new process. He’s stumbled across the means to produce salt by “destroying” the water. Unfortunately 
for the player, an emergency occurs in which the player is trapped without access to drinking water. The 
player must then figure out how to make drinkable water with only saltwater available to them. Bill doesn’t 
believe that this is possible and the player must convince him through argumentation. After successfully 
creating drinkable water, the player survives the emergency. Afterwards, the player is asked to recreate the 
process by which drinkable water is created but on a much larger scale. 


Bill's Grand Idea Water Crisis! 


‘Meet Bill at the Seaside Cave Grab the Pan 


Figure 5: Helping Bill create salt from seawater and collecting evidence from the humidity meter to 
convince Bill that drinkable water can also be created from seawater. 


Unit 6 
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Unit 6 is the culminating experience for our player. There is a planet wide emergency taking place 
and it’s up to the player to figure out what is going on. It seems that water levels are dropping dramatically 
and if the player can’t solve the issue, the planet will no longer be viable for habitation. The player travels 
back to the previous Unit’s locations and takes measurements has to how the water levels in each having 
changed. The player finds that the Sam’s river levels are dropping, Riley’s lake is drying out and the 
humidity levels on Bill’s island are lowering. No one believes your findings though and you must convince 
them that the evidence you’ve collected leads to something extra-natural. Due to the fact that levels are 
dropping planetwide, it must be something regarding the alien’s advanced technology that must be causing 
it. Having convinced the group, the player engages on a fantastical voyage that culminates with them 
transporting to outerspace in order to save the planet. 


—__Moon Base Contral Ro 
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Figure 6: Concept art of the alien walkway needed for teleportation to the planet’s moon and the control 
room for solving the final puzzles to save the planet’s water. 


Argumentation 

A key innovation of MHS is the development of an argumentation engine that does not rely upon 
preselected evidence sets and structures (Griffin, et al. 2016). Reliance upon fairly simple multiple choice 
formats can lead to naive argumentation strategies which are not generalizable to scientific argumentation 
outside the game context, such as using a process of elimination without reasoning from evidence. The 
MHS argumentation system uses pseudo openness and regular expressions to create logic rules for how 
different components can combine. The argumentation engine also allows us to input a number of argument 
scenarios from simple, such as merely stating a claim, to more complex, with multiple evidence items and 
reasoning statements 

We created a user interface with similarities to a solar system with the claim represented as the 
sun, reasoning statements as planets, and evidence statements as moon-like entities for the full system. This 
new structure reimagines the visual representations of connections between claim, evidence and reasoning 
while still adhering to its underlying model. Players are given the largest possible tree structure without 
pre-set drop-zones; allowing students to fill out their solar system as much or as little as they wanted. The 
system therefore allowed us to facilitate pseudo openness and used a simplistic implementation of regular 
expressions to create logic rules for how different components can combine. We then created a priority list 
of all the possible player feedback; so that if a player’s argument matches two of our logic rules; we can 
display the most desired feedback. Further, MHS implements support for understanding and making 
arguments throughout the game play, not just while in the argumentation engine. We highlight claims, 
reasoning and evidence when encountered, provide narrative for framing an argument such as discussing 
driving questions, and provide game activities to help players distinguish between claims, reasoning and 
evidence as well as activities to help players practice critiquing arguments. 

Our implementation of support for learning argumentation in MHS follows the empirically-based 
learning progression suggested by Osborne and colleagues (2013). Their work offers a progress map with 
levels incorporating construction and critique of scientific arguments. MHS follows this progression by 
starting with having students make claims, moving to providing evidence to support claims, and then 
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having students create full arguments. Next students are asked to provide a counter argument to a faulty 
claim as well as having several chances to critique arguments throughout the game play. 


Figure 7: The argumentation engine from unit 5 where the player is convincing Bill that drinkable water 
can be made from seawater. At this point the player has some pieces of evidence, which are not relevant 
and will get feedback to make a stronger argument. 


Dungeon-like Puzzles 


Units 2-5 introduce concepts and have students use their new knowledge to solve a problem, 
typically to help one of their space cadet buddies. We also wanted students to have the opportunity to use 
the knowledge, such as about soil permeability or water evaporating and condensing, in a context where 
they could easily fail and then try again. Adding puzzles to MHS provided an opportunity for putting 
knowledge into practice, alternative explanations, providing natural feedback, allowing players to fail and 
retry, and adding fun. We developed a puzzle schema, which we call dungeon-like puzzles wherein players 
have to move through a dungeon-like space to reach some objective. Movement through the space requires 
solving mini puzzles in each room or floor to advance to the next. For example a player may have to use 
knowledge of soil permeability to regulate water flow to fill a waterway to float resources from one room 
to a next, or use knowledge of evaporation to move water from one area in the dungeon to a next. 


Figure 8: First dungeon puzzle where the player must figure out how to move the power cube to another 
room for opening the water flow and the second dungeon showing the locked pumps the player needs the 
flowing water in order to access. 


Learning Analytics & Teacher Dashboard 
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We employ Learning Analytics to provide monitoring and awareness for teachers and to create 
potential for an adaptive system for student learning and assessment. The analytics track individuals’ 
specific choices, and then analyzes those discrete choices against a backdrop of learning outcomes and 
argumentation competencies; thus assessment is built into playing the game. The analytics populate a 
teacher dashboard to visualize student activity and progress. The dashboard for visualizing student activity 
is designed from a performance support framework to optimize acting upon insights such as recognizing 
when a student is falling behind or getting off track. 
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Figure 9: The teacher dashboard shows a column of students and a a prid for their progress through the 
quests and tasks of MHS. 


Some Key Feedback 


As part of a last design research phase before preparing MHS for an impact study, in the Spring of 
2018 we asked 12 teachers to implement a partial version of MHS in their classrooms. The test included 8 
days of game play for the kids and followup interviews with teachers. We learned a lot from this test which 
confirmed that MHS could be feasibly implemented in classrooms and that teachers were generally quite 
enthusiastic about using MHS with all of them volunteering to be part of any further testing we conducted. 

The MHS implementation had a number of bugs and glitches because MHS was not complete nor 
fully tested against all the operating systems employed by schools for our test. In addition we also found 
that many of the computer systems students used were under-powered for the MHS implementation. This 
teaches us that we need to invest in performance optimization of the game, do more complete testing before 
our impact study and carefully screen schools as well as assist them in using appropriate computers. The 
state of MHS during the spring 2018 test led to more work on the teachers, school tech support, and our 
team as we addressed bugs, glitches and slow downs which impacted the teaching and learning experience. 

While the MHS implementation included some frustration, the students proved to be highly 
engaged. Some comments from the teachers reflect on the classroom experience and kids being focused 
and engaged in MHS play: 


Response: I mean, it made my classroom management for the last ten days great actually. They’re all 
engaged and usually when you have a normal class setting, you’re gonna have kids doze off or going 
out, or so, no. Twenty-five kids working on it for a good 40 minutes of class period. 

Response: Oh, it was actually pretty exciting to see some of my students that normally may not be as 
engaged kind of a little more excited about that. So it was mostly like my boys that are gamers that 
you could definitely tell that they were excited about. I would say that we had almost 100% 
engagement while playing the game almost every day that we did it. It was kind of also interesting 
to watch some of the kids help other students. They were more eager to help and show their 
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knowledge of like, “Oh, you just have to do this.” ....And so there was just a lot of excitement that 
came with playing the game. 

Response: The days went fast. Like even the kids said like the normal 47-minute class didn't feel 
like 47 minutes. It flew by because everyone was moving and talking and working. 


Typically we think of a single player learning game as simply an individual learning experience or 
even potentially isolating. However a number of teachers reported on their classrooms during game play as 
being very social places. One of the comments below reflects a positive social atmosphere that contrasts 
somewhat with the teacher’s experience of how middle school students often interact. 


Response: It was very collaborative. I liked that, and they were leaning over and helping each 
other, and they seemed to be excited and engaged. 

Response: I mean, like some people would say something, “I can’t figure this out.” And they’d get 
up and go over there and say, “Go over there, go here.” Yeah. I’d say I wish it happened more in 
my normal class but no, in the normal class, I know a lot of kids that are gonna say, “I don’t 
understand.” But when they’re sitting there staring at the game, they say I don’t get it, and nobody 
says, “Oh, whatever, you’re dumb.” They went and kind of helped them, so, I guess it gave them 
more of a comfort level and then, those that were done I guess had a feeling of accomplishment, 
hey, I know how to do this, P’Il show you. ... there were kids helping kids that usually wouldn’t 
help kids back in a normal classroom setting. 


One area needing substantial improvement is the student use of the argumentation engine. We 
designed, developed and usability tested the argumentation mechanisms, but in the wild, actual classrooms, 
we found a substantial number of players trying to brute force their way through the tasks. Some students 
had over 50 trials on a single argument. Scientific reasoning and argumentation is difficult for middle 
school students so this starts as a challenge, but perhaps the students found the argumentation tasks too 
tedious in the flow of the game, lacked confidence in their ability to succeed at the tasks, or required more 
reading and thoughtful activity than the players wanted to invest. Subsequent to this testing we have done 
substantial revision to leading up to the argumentation tasks, support within the tasks, improved feedback, 
and improved support for the teacher as she/he seeks to help the students. One teacher summed up what she 
saw in her classroom. 


Response: They got frustrated when they reached the argument stage of course. They wanted to 
answer now. They didn’t really want read to find the answer, so, that was — I think that was the 
frustration level. 

Response: It just became like a manipulation guessing game. Well, I have this reasoning, let me try 
evidence one, two, three, and four. Oh, that doesn’t work, let me try one, two, three, and five. Oh, 
that doesn’t work, let me try one and two. 


While argumentation needed improvement to meet our expectations for student learning, teachers 
reported that students were learning from the game play: 


Response: I had a girl who for two years I've been trying to teach about why the oceans are salty and 
things like that. And all of a sudden, she's like, "Oh, no way! It's salty because water's evaporating." 
And I'm like, "Really? You got it from there from an argument in a game?” Like it was just so cute 
to see them get so excited because they learned something. 

Response: I think it was a fun way to learn science. I think it just made it a lot more interactive than 
me standing up and lecturing, and [students] taking notes. 

Response: this is coming from a perspective from somebody who understands the scientific content. 
Once I saw them figuring it — figuring out things in the game, once I realized, “Ah, this is what 
they’re supposed to do here.” It’s a great way to teach that concept, 

Response: A lot of them were playing the game as if they were actually the character and I guess it 
was almost like first person. First person experience and that they were actually trying to solve the 
problem as if they were the ones who were trapped or whatever, and you feel like, so they actually 
had a reason to try and kind of be in control of their own learning, whereas when I give them certain 
assignments that they’re not that impressed with especially if they’re on paper. They’re like, why 
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are we doing this? What does this have to do with me? That kind of thing. Why do we need to 
know this? 

Response: I wish there was more games like this because it really engages students and it — I think 
it’s more authentic and I think it makes them think more like a scientist. 


Teachers also reported that the game play enabled students who were not typically high achievers to 
engage, achieve and even become leaders in the classroom. 


Response: I had students who, like for the first time that they've been with me, seventh graders, and 
I've had them for two years at that point. They said this was the first time that they felt like they got 
to use their strengths to show that they actually knew science. I thought that was kind of touching. 
Response: Yeah, that was actually really neat cuz I mean, the cool thing about the game was that 
some of the boys that maybe aren’t so good at paperwork really — I mean, they took off with this and 
were like the leaders, and so they got to help other kids when they were having problems, or they 
finished ahead of other people that normally, they’re never the first ones to finish or — so it was really 
cool to see some of these kids that normally, aren’t the leaders. They became the leaders and that was 
neat. And then some of the girls that aren’t really gamers but are more textbook learners weren’t — I 
mean, they’re used to having everything come easy. I think this was a little hard for them because 
they just don’t play video games very much. 

Response: Yeah, one of them is gifted but he’s also autistic and has severe — it’s like his social skills 
and being engaged sometimes. And he has a hard time communicating on paper but he’s super, super 
sharp, and so that really — he was able to really shine during this. So that was really awesome 
actually. He ended up being a really — a leader which was really cool because it gave him a chance to 
go around and talk to other kids too, so. But he helped me. 

Response: I could definitely say that and not only just the SPED kids, but some of my — the normal, I 
hate to say normal, but yeah. Normal kids but that were really kinda low and never seemed to really 
get anything. As a matter of fact, one of those girls, she was the first to finish and then she was bee 
bopping around, guiding people and stuff and typically, she’s the girl that when I call on her, I cringe 
because I know she’s gonna get it wrong. 


Gaming seemed to bring out non-typical school responses. Typically high performing students did well 
in MHS but some struggled as they found gaming to emphasize different approaches than they were used to 
and enabled other students in the class to shine. Differences were also noted between boys and girls as well 
as across grade levels, but not all the differences were consistent from teacher to teacher. 


Response: I had some kids that were so into it, it was pitiful. They were just — they were chomping 
at the bit and I had some kids that, “This is stupid. I wanna do bookwork.” 

Response: These students in our seventh grade that participated definitely interacted vocally but 
more just in their reactions and our eighth graders interacted vocally and physically in a sense that 
they would go to the person next to them and they would help them out, explain things, and kind of 
verbally talk through the problems whereas our seventh grade was kind of more like, “Whoa, look 
how funny that person looks.” And they just kind of found like, “Oh, you have to meet them in the 
corridor or in the walkway through the mountain.” And so it was kind of interesting to see that big 
of a change between the two grades since I mean, they’re in the same setting. 

Response: You could go through and when you’re going through and you’re setting the sensors in 
the river, the kids that really do a lot with their just studying stuff, they would set one or two sensors 
and expect all the data to flow in, and the guys sitting next to them were like, “No, you gotta go all 
up and down the river. Go in the little inlets and all that and see where it is.” So they would be 
helping out in that respect whereas the ones that were really like I said, the book learners, they would 
expect they push this button once and all the data to flow into their mind. 


Next Steps 


As noted in the introduction the MHS team is planning a substantial field test of MHS for the 
winter/spring of 2019. To reach the field test we have prioritized several objectives. First, we need to 
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achieve technical soundness, optimization to perform on as low a performance computer as possible, and 
clear direction for what systems will work. Second, improvements across a range of quests and tasks to 
better achieve learning outcomes. Third, making argumentation a better fit to the rest of the game play and 
providing support for students who are likely to struggle with the competencies. And fourth, support for 
low readers by adding audio for dialog, low gamers by adding better feedback, clearer graphics, more 
tutorials and simplifying some game mechanisms and activities, as well as support for high gamers by 
enriched graphics, rewards and side quests. More information about MHS can be found at 
MHS.missouri.edu or by contacting the authors. 
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Abstract 


This paper summarizes the discussion at The Association for Education Communications and Technology 
(AECT) Conference that was held in Kansas City, Missouri on October 23-26, 2018. The objective of the panel was 
to discuss the current state of being an online graduate student in educational technology and identify future support 
and research directions to support the community. The panel consisted of two faculty members and three students 
from two large universities within the United States. The diversity of the group is within their individual experiences 
and current place in formal education and career. Each panelist was asked a series of questions. They were then 
able to describe the current online environment from their perspectives. Several common themes arose including 
program support, online community built, project based work, and finding balance with school, work, and 
life. These are discussed in turn along with future directions identified by the panelists. 


Introduction 


Across the United States, colleges enrolled more than two million students exclusively at a distance in 2014 
(Allen & Seaman, 2016). Many of those students are graduate students as many institutions prefer to start offering 
graduate programs at a distance due to less required curriculum credits (Allen & Seaman, 2016). The popularity of 
online programs could be attributed to adult learners wanting more flexibility in learning environments and being 
ready to access education (Ilgaz & Giilbahar, 2015). 

To gain a better understanding of how online graduate programs in educational technology are viewed and 
being implemented by research institutions, a panel discussion Portrait of a Modern Online Graduate Student in 
Educational Technology was held at The Association for Education Communications and Technology (AECT) 
Conference in Kansas City, Missouri on October 23-26, 2018. The objective of the panel was to discuss the current 
state of being an online graduate student in educational technology and identify future support and research 
directions to support the community. The panel consisted of three students and two faculty members: (1) Anna R. 
Leach from The Ohio State University, (2) Cara A. North from The Ohio State University, (3) Natalie R. Gintert 
from The Ohio State University, (4) Dr. Ana-Paula Correia from The Ohio State University, and (5) Dr. Leanna 
Archambault from Arizona State University. Each panelist was asked to discuss why they chose educational 
technology, identify challenges in programs and provide insights into building online community and the future of 
online programs. 

The discussion began with each participant sharing their unique backgrounds in formal education and how 
it has shaped their perspectives. They also shared a little about their personal background to provide context to what 
they manage outside of the online program. The student panelists also discussed the challenges of balancing their 
outside life responsibilities with those that were specific to the online classroom. They described how projects 
brought them together and created transferable skills, outlining various projects such as one that required working 
with a client internal to the university who needed aid in building an online learning environment and improving an 
existing online course. The project took the entire semester and included a needs analysis, multiple reports, and a 
final presentation to the client. Another class project involved evaluating a learning tool and consisted of a context 
report, a pilot plan and tryout, and two live class presentations. The skills built from these projects were transferable 
skills such as the ability to work with a client, present at a conference, and create innovative learning 
environments. In addition, the panelists also reviewed the practicum process that is the culmination of the master’s 
program. The practicum is an applied educational technology online course that consists on a 12-week project with a 
real-world client. The work in this class is structured in different types of deliverables and overseen by the student’s 
advisor. It culminates with a presentation to the client and the student’s committee since this is the program 
culminating project Students shared how the practicum project and other portfolio activities impacted their world 
outside of the online courses and after graduation. Being able to handle the large workload of online courses while 
keeping up with professional and family demands is a unique challenge, and the panelists shared their experiences, 
struggles, and strategies when it came to managing it all. 


The faculty members shared their universities’ program structures and describe the expectations for the 
students. Faculty described specific support structures, including online discussion forums designed to get students 
involved before classes begins as well as efforts to offer them a programmatic well-round and rich learning 
experiences. Strategies for keeping online students engaged, not only in class, but in the online program as a whole 
were discussed. Supporting graduate students in an online degree program as individuals and understanding their 
complex needs, as learners, parents, and employees, among a host of other roles, was a key topic of discussion. 
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The remainder of this article summarizes the discussion that occurred at the panel and is intended to serve 
as a record for the online graduate student community that were unable to attend. 


Please introduce yourselves and share your background for the audience. 


Anna Leach: 

I am currently a PhD student in Information at the University of Arizona. I finished my Master of Learning 
Technology degree online through the Ohio State University College of Education and Human Ecology in 
the fall of 2017. While working on the master's degree, I worked part-time, remotely from Tucson Arizona 
for the Ohio State University as a data analyst. I returned to school in 2016, 12 years after completing my 
Bachelor of Arts in Mathematics in a traditional face-to-face program. I have had online classes before 
being accepted into the masters program, but only one at a time and only occasionally. I am also a wife and 
mother of 2. 


Cara North: 

I am currently a PhD student in Educational Studies with an emphasis in Learning Technologies at The 
Ohio State University. I started in the Master of Learning Technology degree online through The Ohio 
State University before transferring into the doctoral program. I am a full time learning experience 
designer for the College of Education and Human Ecology at The Ohio State University and have worked 
in learning and development for all of my professional working career. I’ve worked for various non-profit 
organizations and as an instructional designer for Amazon.com. I’m also addicted to volunteering and 
serve as President-Elect of the Central Ohio Chapter of the Association for Talent and Development, 
Founder and past-President of Toastmasters International at The Ohio State University, Community 
Manager for the Training Learning and Development Community, and was recently elected to 
Communications Officer for the Research and Theory division of AECT. I also co-host a learning and 
development podcast called Instructional ReDesign with my good friend Joseph Suarez. 


Natalie Gintert: 

I just recently graduated with a Masters in Learning Technologies and work full time as a Program 
Manager. I run training programs that teach mid-career professionals leadership management skill. With 
the use of my degree, I’m currently working on implementing online learning into one of our credit barring 
programs - The Certificate in Public and Nonprofit Leadership. 


Dr. Ana-Paula Correia: 

I am an associate professor in learning technologies in the College of Education and Human Ecology at The 
Ohio State University. Recently I have started a new leadership role as the director for OSU’s Center on 
Education and Training for Employment. I am currently the faculty co-coordinator for OSU’s master of 
Learning Technologies. I share this role with my colleague, Rick Voithofer. I have more than 25 years of 
experience in learning design and instructional systems technology. Specifically, my expertise in distance 
education, online and mobile learning, collaborative learning and entrepreneurial educational approaches 
have been published in many academic journals. In 2017 I co-authored the book, Teaching Online 
Simplified: A Quick Guide for Instructors with my former doctoral student, Farrah D. Yusop. At OSU I’ve 
founded the Learning and Experience Design Research Group, a group dedicated to investigating learning 
design processes and pedagogies to create deep learning. I am an active contributor to the AECT, serving as 
President (2011-2013) for the Research and Theory Division. 


Dr. Leanna Archambault: 

I am an associate professor in learning design and technologies in the Mary Lou Fulton Teachers College at 
Arizona State University. I am the program coordinator for ASU’s Masters of Education program in 
educational technology. My research areas include increasing sustainability literacy among pre-service and 
in-service teachers, teacher preparation for online and blended classrooms, the use of innovative 
technologies to improve learning outcomes, and the nature of technological pedagogical content 
knowledge. I’m the Co-Editor of the Journal of Online Learning Research, focused on K-12 online and 
blended learning, and I am the Associate Chair of the Information Technology Council for the Society for 
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Information Technology and Teacher Education. I am also co-PI on a grant funded by the U.S. Department 
of State, to provide technology-related professional development to international teachers during a 
semester-long residency at ASU. 


Why did you choose to be a part of an online masters program in the field of educational technology? 


Anna Leach: 

I have always enjoyed the idea of training or teaching people to become more skilled in their jobs. For 
example, as a data analyst, I had the opportunity to train fellow university staff members on the student 
information system. Teaching them how to use MS Access to gather data about their populations so that 
they could make data driven decisions. I love the expression on people’s faces when they realizes what 
they can do. However, asking staff to sit in a face-to-face classroom for 8+ hours can be difficult and 
retention of the information is unlikely. We recognized that it would be much easier for the person to be 
able to have the information readily available to them for their own consumption. This is where technology 
can and should be utilized. With this real-world example and my desire to start pushing my career into the 
teaching and research arena, | started to explore programs; educational technology was an obvious choice. 


Cara North: 

I fell into the profession of learning and development after I started my professional career in a call 
center. Since then I have worked in corporate and higher education settings in learning and development. 
Seeking a degree in educational technology was important for me to ensure that my on the job knowledge 
wasn’t full of holes. I appreciate the theories and history that I’ve learned throughout the program and I 
can see how it’s made me a thoughtful designer. 


Natalie Gintert: 

For me, I was looking for a degree program that merged my work experience and my bachelors degree. I 
received my undergraduate degree in Electronic Media Productions, and ended up working in Higher 
Education in training and professional development. So, I really saw educational technology as a way to 
merge my skills. Additionally, I wanted to learn more about the theories behind online learning, 
specifically how do you build communities well online. I felt like it was just a natural fit that the program 
that was teaching you about using technology in education used technology to deliver the learning online. I 
found a few degree programs that were teaching educational technology, but were not offering course with 
the use of technology. So, I end up in the program I graduated from because I appreciated that it was 
talking the talk and walking the walk. 


Dr. Ana-Paula Correia: 

When I started my position at The Ohio State University I was pleasantly surprised and excited to find out 
that I would be teaching exclusively online to a newly created online program in the learning technologies. 
The master of Learning Technologies (MLT) is a professional degree that focuses on applying current 
research to solving problems of practice relating to the use of technology to support learning in a various 
in-person, online, and blended contexts. Students in the degree take a core sequence of seven courses, a 
research course, one elective course, and a practicum course, completing 30 credits in total. This was the 
perfect opportunity to continue researching about online learning and teaching while bringing to life many 
teaching strategies that I had tested and validated in previous online courses at a different university. I 
instantly embraced this opportunity. 


Dr. Leanna Archambault: 

After two major college reorganizations, it was necessary to combine one masters in ed tech focused on K- 
12 teachers and another masters in instructional design focused on the workplace/higher education. We 
came together as a faculty and decided to move the program to an online format to be able to target a 
national or even international audience. The program emphasizes the design of learning technology 
environments, systems, applications, and instructional materials for multiple learning contexts. The 
program prepares learning design and technology professionals for a variety of settings, including business 
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and industry, and for formal and informal educational settings, such as K-12 and higher education, 
museums, and nonprofit, and government organizations. 


What do you believe are the greatest benefits of the online masters program? 


Anna Leach: 

Flexibility and reachability. As a parent, it is difficult to be able to find care for children around a school 
and study schedule. Online learning allowed me to attend class with a sick child on my lap or while in the 
audience of taekwondo practice. I was able to keep my family as a number | priority while still reaching 
for my goal. The flexibility of the online program made the impossible feel possible. Second, I say 
reachability because of the people that can be reached. I enjoy meeting new people and gathering 
perspectives. There were people from all over Ohio in our program! The insights brought to the program 
made it a richer experience. 


Cara North: 

Online education allowed me to control much of my learning. If I wanted to dive deeper into a topic, I 
could and often bringing those insights into course discussions were valued. I appreciated seeing many of 
the instructors wear a facilitation hat versus being a sage on the stage. This allowed for more authentic 
learning experiences. 


Natalie Gintert: 

I found the online format to be easy to fit into my everyday life. I’m a working professional and my days 
are typically mapped out from 8:00 am to 5:00 pm. So, I really loved that I could schedule my education 
into my busy day. I could look at my schedule and when I had 15 or 20 minutes I could take a break from 
work and focus in on learning. I didn’t have to wait until class time to learn something new. A Lot of 
times I would learn something in the course that I could apply right away at work. There was no, taking 
notes and trying to remember to implement it later. I had that instant connection to how to make this 
practical. 


Dr. Ana-Paula Correia: 

One of the greatest benefits of an online masters program, from a faculty perspective, is the opportunity to 
offer a high demand “product.” Such is going to meet the needs of many professionals that are eager to 
advance on their careers or go after a career shift. Additionally, when starting a new online program, 
faculty and staff have the responsibility to create and maintain high quality online learning and teaching 
experiences. One aspects of the master of Learning Technologies that our team at The Ohio State 
University is particularly proud is the Practicum experience. In every course, students create learning 
artifacts that are included in a portfolio that is developed as part of the Practicum. However, the biggest 
strength of the Practicum experience is to work with a real-world client, identify an instructional problem 
and address that problem with a tangible solution. The Practicum is a structured online experience carefully 
guided and supervised by the student’s advisor. It includes five distinctive milestones, as for example, 
establishing a memorandum of understanding and regular progress reports to the advisor and client. 


Dr. Leanna Archambault: 

I think one of the greatest benefits for our students is the flexibility afforded by the online format. Our 
students take 1-2 courses in a 7.5 week format. This allows them to focus intently on course content, while 
juggling their work responsibilities. They can complete the degree in as little as 18 months or if necessary, 
take one course at a time and go at a slower pace that would allow them to better balance work and family 
obligations. It also allows for rolling enrollment, so students can start the program in either in the fall, 
spring, or summer. Many students have indicated that without the online format, completing a master’s 
degree would not be possible. They need to be able to continue to work and care for their families, so 
online courses are realistically the only viable alternative. 


In addition, the online format allows us to attract student from a vast array of contexts and backgrounds. 


Given the diverse nature of backgrounds, students have the opportunity to work with one another and share 
their knowledge and understanding, applying design to various contexts. 
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What are the challenges of an online masters program in educational technology? What strategies have you 
utilized to overcome these challenges? 


Anna Leach: 

Time management was and is a big hurdle for me, but I also feel the sense of isolation can be an issue in an 
exclusively online program. With respect to time management, it was important for me to know what was 
expected of me as soon as possible with respect to the program and the individual courses. Trying to plan 
workload around work and family is critical to successful time management. It is better for me to know 
what the schedule is and important due date even before the class began. It was also crucial for 
communication between the instructor and myself to be timely. To overcome or avoid a sense of isolation 
in the program, I had to recognize the community that was available to me. The discussion boards, 
although they can be underutilized or not utilized properly, were a place that I could read others work, but 
the synchronous sessions in which students either presented or spoke or chatted were really were I felt a 
sense of community most. These chance to learn together and have timely conversation helped the 
program feel like a community of learners instead of isolated learning. 


Cara North: 

My biggest challenge is balancing school in addition to being a full-time learning experience designer. It 
has been difficult to sacrifice time away from my family and friends but I keep telling myself it will be 
worth it. I also struggle with making myself do tasks that I find boring or not applicable. For example, 
despite being an instructional designer for ten years, I still had to take an introduction to instructional 
design. I wish I had the opportunity to demonstrate the competencies in the course to test out of it. I found 
I was disengaged in the course but took the assignments to push my skills more. For example, despite the 
course wanting a report, I created a report and a digital visual aid in a tool I hadn’t used before so I could 
learn a new skill. While I was motivated to do that, I fear many online graduate students aren’t. 


Natalie Gintert: 

I found it difficult to accomplish the weekly discussion posts. Sometime, I felt like assignments were just a 
way for the instructor to make sure I read the materials. Not to see if I was actually learning and 
understanding the concepts. Is a discussion prompt always the best way to see that a student is 
understanding the materials? I would argue no. I think we need to think about how students can start to 
apply the concepts as soon as we give them to them. Not just discussing how they “felt” about them. But 
having them think about how they can use the in their life and work. 


Dr. Ana-Paula Correia: 

One of the biggest challenges to overcome in an online masters program is the feelings of isolation and 
disconnectedness as a learner. Feeling disconnected from their peers and instructors as well as from the 
program, department, college.... Another important challenge is not having their unique skills and 
knowledge as professionals in specific fields valued or even acknowledged. As adult learners (Knowles, 
1996), these students have much to offer. For example, adults want to know why they should learn 
something. Just to say this is good for them to learn is not enough. A reason for learning something needs 
to be offered. Adults have many life and professional experiences that they have accumulated (e.g., work- 
related activities, family responsibilities, and previous education). While teaching, instructors need to 
connect learning to this knowledge/experience base and use it whenever it is relevant to the topic. Adults 
are goal-oriented. Upon enrolling in a course, they usually know what goal they want to attain. They, 
therefore, appreciate an educational program that is organized and has clearly defined elements. Instructors 
must show participants how this class will help them attain their goals (Yusop & Correia, 2017). 


Dr. Leanna Archambault: 

One of the challenges we have faced, in addition to those already mentioned, is being able to structure 
longer term group projects given the 7.5 week format. It’s a fast pace, and while there are a number of 
advantages, one of the challenges from a design perspective, is getting students organized into groups. 
They struggle trying to coordinate across time zones and work/family schedules. We have tried to address 
this by better aligning content across courses so that projects can have a longer term. We have also been 
mindful to make sure the learning outcomes indicate a need for group collaboration and have been strategic 
about where we incorporate these projects. 
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From an administrative perspective, we have a lead instructor for each course (who is typically a 
tenured/tenure-track professor) and then a number of academic associates (AAs) who each have a group of 
students for whom they are responsible. This structure works well with academic associates who have 
experience, but it can be a challenge to attract and retain them, given the nature of the supplemental work, 
so that is an ongoing focus. Making sure that the AA’s stay on track and that there is a close working 
relationship between the lead instructor and the AAs is important. 


What are systems and support structures from a program and/or university level that can be helpful to 
graduate students in completing the program. How can faculty help support online graduate students? 


Anna Leach: 

My favorite part of the program was the opportunity to present the information that I learned. Not only did 
my professors have facilitation as part of the curriculum, but they also pushed us to present the materials 
from projects at conferences. Prior to this program I had never considered a class project to be material 
worthy of presentation, but their suggestion gave me confidence in my abilities. Because of that, I have 
presented with my group and met many driven individuals which has grown my community further. I have 
also made myself more marketable in the profession and to the PhD programs that I applied to. 


Cara North: 

One of the best ways for faculty to support online graduate students is to get to know them and their 
strengths and areas of opportunity. I knew I wanted to grow in visual design and analyzing data and I was 
able to work with my professors to target assignments to challenge me in those areas. 


Natalie Gintert: 

I think providing opportunities for students to apply their new concepts is key for a successful program, 
online or in person. I think sometimes academic programs get too caught up in making students good 
writer, which is good. But, how can I translate the 5 research analysis papers that I’ve done into skills I can 
use for a job. Sure, I know how to think critically and I have an understanding of current research in this 
area, but do I have anything where I can show an employer “this is what I’ve done and here is the system I 
used to create it”. 


Dr. Ana-Paula Correia: 

I order to support online students, faculty needs to establish a frequent and consistent communication 
system with their students . Not only for checking students progress towards graduation, but as importantly 
to build and enhance their confidence as online learners. I believe this instructor presence needs to be 
particularly strong on the first weeks of the course to establish a working relationship. Even prior the class 
starts I get in touch with the students enrolled. I always send a welcome email that spells out the 
expectations for the first two weeks of classes. Also if there are any live sessions during the semester I let 
students know the dates and times so they can make arrangements to attend these sessions early in the 
semester. I use chat and video conference to stay in touch with my online students. The message is that we 
are in the learning journey together and that I am there to help and guide them towards success. 


Dr. Leanna Archambault: 

It’s important for online graduate students to know that there are many resources offered by the university 
that also are available to them, including writing center support as well as counseling services. Making sure 
students are aware of these supports often falls to faculty to share/encourage students to seek them out. 
Often, getting students simply to reach out to the instructor takes some encouragement, particularly when 
“life happens.” I was encouraged to read of Reesa-Marie Dawkins’ statement included in her syllabus that 
reads, “When life happens, send me an email... can give extended grace periods, tutor you one-on-one by 
phone, be a good listener, offer a list of campus resources, and help you catch back up, if you have fallen 
behind” (Supiano, 2018) I try to take a similar approach, but I have to emphasize to students that they need 
to reach out and let me know so that I can help. Being approachable and student-centered is essential when 
teaching in an online program, and given the hectic nature of all of our daily lives, it’s imperative for 
faculty to be understanding and to work with students as issues arise to help ensure their success. 
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How important is the online community in the program? How did you build your online community? What 
supports has your community provided? 


Anna Leach: 

The online community was critical to my success. As we have all mentioned, we learn from each other and 
our faculty. Their support and the ability to be supportive is the right environment for learning. Dr. 
Correia taught me to put our work out in the open, to be shared. She introduced me to the AECT 
conference and Dr. Savenye; who then introduced me to Dr. Archambault. Cara introduced me to twitter 
and an entire world of instructional design professionals who apply the research. And for the first 
presentation at a professional conference, Natalie and Cara were by my side. My community has brought 
me intellectual and emotional support. As I said before, I had a fear of feeling isolated, but this community 
kept me grounded. 


Cara North: 

I would be nothing without my online community. I often tell peers that it is easily the number one mistake 
I’ve made in my career as a student and professional. Network, network, network. I learned this lesson the 
hard way when I moved to Columbus, Ohio and left my job as an Instructional Designer of 5 years for 
Amazon.com. While working at Amazon, I had a balanced team of designers and editors and I didn’t 
realize how valuable they were until I went from a team of 40 to a team of 2 in my role at The Ohio State 
University. While I could still network with my former colleagues at Amazon, I found they spoke so much 
corporate jargon that it was not beneficial to me as a learning designer. In 2016, I met a learning and 
development professional through a local professional development society, The Association for Talent and 
Development. He changed my life by telling me that Twitter was the number | professional development 
tool for learning and development professionals. I was so engaged by what he said, I decided to sign up for 
Twitter. Two years later, I have a network of learning and development professionals from all over the 
world. I like to say that when you work with me, you don’t get me but you also get my network. My 
community has provided me a wealth of knowledge and helped me identify opportunities for leadership and 
engagement. The best part is that I have a passion of helping others connect and I try to pay that forward 
by offering opportunities to others. 


Natalie Gintert: 

As Cara stated, the community was everything for me. I would not be as enthusiastic about online learning 
or educational technology without my community. I learned so much from the students I actually 
accomplished projects with. In our program we worked through a large project with an outside clients, so it 
was a really easy way for me to learning how to actually “do” it in a low risk environment. At the end of 
the day that will be what I remember the most, the people and what I learned from them. I know that I 
could reach out to any of my team members and ask them for some feedback and have a better product 
because of it. 


Dr. Ana-Paula Correia: 

The online community in the program is critical to everyone’s success. To the students’ success, but also to 
the faculty, staff and the program in itself. One strategy that I use is creating a space and opportunities for 
getting to know each other. There are multiple ways to get to know your students online. For example: 

Use a discussion board on the learning management system you selected to create a 

discussion thread titled “Class Introductions”. Ask each student to introduce themselves in the discussion 
board by stating their name, gender, current job or working experience, country (for international students), 
etc. Ask students to post photos of themselves, their families, pets and/or place where they live. Most 
international students will be proud to share information about their countries and customs. Encourage 
students to us audio segments and video clips in their introductions. As the course instructor you should the 
first to post your introduction on the discussion thread using different types of media to model what is 
being asked from the students. A virtual cafe is set-up in class to allow for social interactions throughout 
the entire course. This year we have invited faculty and students for an happy hour that is going to take 
place at the student union where remote students and faculty can join via videoconference. This is a 
program level initiative to support community building. 
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Dr. Leanna Archambault: 

The online community is absolutely essential for students in an online masters program. As faculty, we 
make connections, but then encourage students to build upon these introductions and extend them, creating 
a vast support network. Particularly in their first core course “Foundations of Educational Technology” 
students create a multimedia “All About Me” presentation using any medium of their choice. After this 
assignment, students quickly form bonds and enjoy having one another from one course to the next. We 
hold informal “Coffee Chats” online where students can come by and talk to faculty and advisors. We also 
have an Ed Tech listserv that students subscribe to. This allows them to stay connected with program 
alumni, many of whom are now professionals in the field, and are looking to hire from our recent program 
graduates. This listserv has served as an excellent source of online community, particularly when it comes 
to job or internship placement. 


How do you envision online graduate programs to evolve in future? What will your online program look like 
in 2025? Will there be a need for it? 


Anna Leach: 

I expect them to keep improving, but yes, there will always be a need for them. I expect them to utilize 
new experiences and opportunities, in combination with new and existing technology, to create better 
learning. I expect them to continue to push their students to be innovative. I expect the online educational 
technology program to be the best online program. 


Cara North: 

What is old is new again. The future of online graduate programs for me is to model an 

apprenticeship. For example, in educational technology often students learn about theories and lack real 
world experience to build upon those theories to build learning artifacts that could be used in a higher 
education or corporate setting. Students in online graduate programs may struggle with making 
connections to internships and work opportunities so having a system in place like a matchmaking 
application to pair work with capable students would also be a great advance that would result in more 
win/win opportunities. 


Natalie Gintert: 

I’m hoping my 2025 not only is technology more advanced, but our thinking about how to use the 
technology is more advanced. I think there will always be a need for online programs, people will always 
be busy. But, I’m hoping we moved away from checking an online box to more project based... 
application of a concept. 


Dr. Ana-Paula Correia: 

There is going to be a need for online programs in 2025, but current online education needs to evolve into 
rich and authentic learning experiences. For example, internships, employment during college, 
volunteering, and extra-curricular activities provide opportunities for people to showcase how they apply 
their knowledge and skills, make decisions, and create solutions in real working situations. As part of the 
master of Learning Technologies, we have an optional | credit course that online students can enroll to be 
part of a virtual internship opportunity. The purpose of this course is for online students to have the 
opportunity to engage in real-work experiences in a variety of workplace settings. These experiences do not 
require the students’ physical presence in the internship site as most of these will be supported by 
technology and are unpaid. The goal is for students to strengthen their professional portfolio and augment 
their programmatic experiences, especially at the master’s level. These experiences are mainly in 4 to 6- 
week projects and may consist of: (1) participating in the development of an educational product through 
various activities (e.g., conducting a needs assessment, reviewing various prototypes, assisting with the 
design); (2) assisting with design and development of training modules; or (3) assisting leadership on 
making learning technology-related decisions. 


Dr. Leanna Archambault: 
I think technologies that facilitate learning will continue to advance and we will see increased use of 
personalized learning in the masters program. There is certainly a need for the program to continue to 


333 


evolve to better meet the needs of students. Currently, we only have one elective in the program. We may 
want to expand to allow for additional student choice in the masters program. In addition, expanded 
internship opportunities prior to the applied project, to allow students more real-world work experience 
may be another direction to pursue. Across so many industry areas, we see a growing awareness for the 
need for instructional design, so making sure our students have the necessary skills to hit the ground 
running is always a primary focus. This ensures the necessity for our program, but we need to be willing to 
make adjustments to make sure the degree continues to be relevant. 


The panel presentation shared the challenges and rewards of participating in an online master of educational 
technology program. Each institution has constructed ways to support the student for success and suggested ways 
that the program can do better. Each panelist conveyed that time management and organization are keys to success 
online. They also believe that progression in technologies will help facilitate learning. Finally, they emphasized the 
need to create rich experiences for authentic learning. 
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Abstract 


The paper discusses the feasibility of punctuation recognition in oral delivery through voiced and unvoiced 
speech detection. The paper is firstly focused on introduction of iB-CET, in an attempt to clear away possible 
illiteracy of linguistic perception and speech recognition technology. Secondly, the paper providess some touches on 
how linguists interpret punctuation, thus inducing the third part in interpreting engineer’s concept on punctuation 
relying on Bark wavelet transform parameters and subspace analysis in speech punctuation recognition. Finally, 
through experiments, the paper convinces readers that punctuation in speech can be accomplished, hence leaves 
much thought for the notion of punctuation recognition in speech for oral delivery. 


Keywords: punctuation; voiced and unvoiced; 1B-CET 


I. Introduction 


The present Internet-base College English Test (B-CET for short) is getting popular, though some scholars 
are still arguing against its credibility and feasibility. Designed and planned in May, 2007(Jin Yan, & Wu Jiang, 
2009), the iB-CET got started among 53 colleges and universities in 20" of December, 2008 so that China made a 
milestone in computer-based test rather than paper-based test for high education in history(Liu Jian-gang, & Dong 
Jing, 2011). The main feature of iB-CET aims at oral delivery in the exam. This marks the very breakthrough in oral 
test to paper-based CET, owing to the contribution of speech recognition technology based on auto-computer scoring 
system. The oral test takes a proportion of 10% in the whole test marks in iB-CET, making a revolution in the 
paper-based test system, hence the iB-CET system comes into life, a wonder in Chinese education for English 
teaching. Anyhow, for whatever reasons and senses, the design and plan worked and became accepted by the 
authorities and faculties in educational fields(Jin Yan, & Wu Jiang, 2009). 

However, the most challengeable argumentation results in the credibility of speech recognition of test 
contents in oral delivery, because nobody can find a satisfactory explanation on the technology and software design of 
speech recognition model in oral delivery on the official Website of CET(www.ccets.org). All doubts come from the 
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ignorance of linguistic perception and speech recognition on oral delivery detection model of the target language, 
leaving much to be explained by linguists and computer engineering scientists. 

This paper attempts to clear away possible illiteracy of linguistic perception and speech recognition 
technology through an illustration by oral test scoring method with recognition on punctuation in voiced and 
unvoiced sound in oral delivery, trying to convince people involved that iB-CET could work in some ways. 


II. Linguistic Perception on punctuation 


Linguists feel strongly the movement in higher education to replace or supplement traditional pedagogical 
methods (eg paper-based or face-to-face learning) with online learning has seen considerable acceleration in the last 
few years, especially in relation to distance learning(Lisa Emerson, & Bruce MacKay, 2011). In traditional 
pedagogical methods, students can easily respond to teachers according to the contents in written form, deliberately 
noticing all kinds of punctuation while uttering in perfect pauses, making no syntactic or semantic mistakes. For 
example, in paper-based oral test, students are usually required to read the text as to be tested about their fluency and 
accuracy in understanding of the content. Hereby, we have to notice that students can make perfect contextual 
understanding of the reading material with assistance of punctuation such as comma, full stop or exclamation marks. 
Since the material is entirely text based, the students can easily take care of the structure of the content, making a 
perfect answer to examiner, which leaves us with a conception that punctuation is very important in syntactic and 
semantic understanding of the material in paper-based test. So, a question emerges into people’s mind what is the 
role of punctuation in learning and test? 

Throughout its history, punctuation has been employed for varying purposes(Robert J. Scholes, 1990). 
What was originally used to signal breath breaks in reading aloud has in the course of time come to signal for the 
most part division into structural units in both silent reading and reading aloud, thus serving to disambiguate 
meaning and provide clues for coherence of the written. Punctuation in the written is not an adequate substitute for 
prosody in spontaneous spoken discourse; punctuation has, therefore, only a remote relationship to prosody. 
However, in the case of reading aloud, the relationship is much closer. And even in the case of written texts 
themselves, punctuation can be used to simulate spontaneous spoken discourse, “What the hell!” for example. 

We know that the written was originally without punctuation; such a script is formally designated as 
scriptio continua (continuous speech). The last chapter of James Joyce’s Ulysses, Molly Bloom’s soliloquy, is 
written almost entirely without paragraphs, capital lettering (except for proper names), and punctuation. The text 
thereby becomes extremely difficult to read, whether silently or aloud(Daniel C. O’Connell, & Sabine Kowal, 
2008). This can be well met with the ancient Chinese book “Origin of Chinese Characters”. 

Hereby we come to a question: in what way can we define punctuation? We can define it in elocutionary 
and syntactic functions. In elocutionary function, punctuation serves as a set of instructions for reading a text aloud, 
more specifically as one aspect of written speech(written form and a kind of phonetic transcription(sound form) for 
prosody (stress, pause, and intonation). In syntactic function, punctuation serves to convey meaning. It does so by 
identifying lexical elements and clausal, phrasal, and sentential structure(Robert J. Scholes, 1990). In these two 
functions, we can then logically come into the definition of punctuation: punctuation is interpreted as notation for 
breathing. Merriam-Webster’s Collegiate Dictionary(11"ed., 2003, p. 1009) defines punctuation as “the act or 
practice of inserting standardized marks or signs in written matter to clarify the meaning and separate structural 
units”, and “to break into or interrupt at intervals”. The Modern Chinese Dictionary(1“ed., 1978, p.69) defines 
punctuation as the written form marking sentence reading and intonation. The Usage of Punctuation (National 
Standard T) established by the Chinese Standard Bureau claims a precise description that punctuation serves as a 
written from of records, punctuation is used for the purpose of making pauses, indicating intonation and sentencing 
or paragraphing. 

So linguists will easily come into a conclusion of punctuation that in our times, every sentence is made of 
words and a specific punctuation mark. A sentence does not make any sense without a punctuation mark. This can 
be perfectly demonstrated in early Greek texts of the fourth century BC, where punctuation was employed for 
paragraphs that marked the beginning of a new topic. Latin documents of the sixth century, for example, employed 
no critical signs, marked neither words nor sentences. The Usage of Punctuation (National Standard T) established 
by the Chinese Standard Bureau defines a sentence as “a linguistic unit making sense of expression through 
intonation with pauses in front and end”. Form the description, we can get the message that punctuation is highly 
related with sentence and intonation. Though there is none special punctuation mark for intonation, still we can feel 
it by the function in punctuation marks of full-stop, question and exclamation. 

In conclusion, when we define punctuation, we have to take into consideration of diction, sentence, 
paragraph, logic, syntactics, semantics, pragmatics and intonation as well, making sense of easy comprehension. 
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This is very practical concept of linguistics, which is fundamental for applied study and scientific research. As for 
the computer processing, engineers can also follow the concept of linguists that punctuation marks be detected by 
voiced and unvoiced speech based on syntactic analysis, semantic analysis, pragmatic analysis and emotional 
analysis. How do engineers follow the concept of linguists? 


III. Engineering Perception on punctuation 


Two scholars Reeves and Nass, from Standford University, found that people treat modern communication 
media as if they are human beings, so established principles of interpersonal communication also predict human 
responses to computers and television(Zhao Li, 2009). Before phonograph was invented, people could only keep 
their memory on paper, called written words. Things changed when people invented recorder, and people can keep 
their memory in sound, called signal speech which can be divided into the voiced and the unvoiced. 

In auto-translation research, computer engineers employed corpora in written words to build up natural 
language processing systems so that machines can intelligently find the punctuation through sentence and paragraph 
boundary from perspectives of syntactics, semantics, pragmatics. Mainly for the sake of the text-to-speech system, 
in early stage, people found the development of a sentence boundary disambiguator(Romportl J., Tihelka D., & 
MatousekJ., 2003). Later, people designed an appropriate classifier where the essential task is to set up its internal 
parameters when they began to make sure of sentence boundary detection. For instance, B. Say, & V. Akman 
illustrated an example in their paper(1996) that punctuation can be served as a boundary detection device as follows: 

He was happy. He found his book. 
(He was happy to find his book. / He was happy because he found his book.) 

Meanwhile, during 1976-1986, come computer scientists undertook punctuation detection on written form 
based natural language processing on the probabilistic analysis of a large corpus. They treated all sentence delimiters 
plus non-letter and non-digit characters as specially-marked, individual words which might have features and 
referred to by constraints. In this way, punctuation marks are used to detect clause boundaries or lists of similar 
categories. Also, in recognizing subjects, punctuation marks such as dashes to the left of a finite verb dramatically 
decrease the probability of the preceding word to be a subject. In the corpus studies, of all the finite verbs preceded 
by a punctuation mark, less than 5% have been found to have the preceding word as the subject. Thus we enjoy the 
fruit of translation soft-wear all around the world. 

In recent decades, people are rushing into the research on punctuation based on sheep recognition in oral 
delivery. This is the follow-up of punctuation based on written form. Engineers use different ways to accomplish 
their goal, such as statistic language model based on Bintree and voiced or unvoiced speech detection. 

1. Statistic language model based on Bintree(Qian Yilil, Xun Endong, & Song Rou, 2006) 

Bintree is based on Statistic language Model to realize its task. In mainly analyzes a sentence into two in 
matching with punctuation. The practice takes it for granted that every word i is among all the words w i (l<i <n) 
on assumption of W = w; w»...... wn, when the sentence is already divided into separate words. Bintree can show 
that between each word wi-; wi, there is a potential linguistic pause, thus for every sentence there are n words 
consisting n-; possible pauses. These pauses from left to right might be pos, pos € (In —1). If a pause mark or 
notice is inserted into the potential pause point, say comma “,” in form a new sentence in W’ = w; w2...wi-7 A wi... 
wn, based on language training model, we can meet the function to figure out the probability on pauses in the 
sentences P(W’) = P(wi), P(w2 | wi)... P(wi |wi-iA)...P(wn|wn-2 wn-11), which means finding out the best 
possibilities arg max pos P(W’). 

2. Voiced and unvoiced speech detection 

There are very few articles talking about punctuation detection on sound form ( speech). Speech pauses ale 
considered as punctuation marks of spoken language. Lots of linguists take speech pauses as voiced punctuation 
marks, indicating intervals in speech. Computer engineers have been making full use of sound corpus, especially of 
voiced or unvoiced detection. The following table indicating the characters of punctuation marks(Table | characters 
of unvoiced speech!) 


' Supported by The National Natural Science Foundation of China: programs of 60572159, 60573184, 60473139. 
the experiments are undertaken with more than 3,000 sentences from sound corpus (slow speed for 2.59 syllables /s, the 
average length for each syllable between voiced and unvoiced takes 114. 21ms. 
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Table 1 characters of unvoiced speech 


Punctuation marks 


eharactas Punctuation marks in 
Punctuation marks in end middle 
iL ! N ’ 3 : 
. 1315- 513- 433- 106- 165- 566- 404- 
Pength coverage otunyoicedi(ms)’ = 5994 2800 3311 1481 2277 ~=S«2764.~—Ss:2803 
Average length of unvoiced (ms) 1794 1088 1132 628 738 1066 882 


Therefore, we can draw a conclusion that we, at present, can arrive at a very practical method to realize 
sound-based punctuation according to table 1 by matching punctuation characters. For the very reason, the paper 
puts more touches on the solution of voiced and unvoiced speech. 

For the detection on voiced and unvoiced speech, there are several methods to get accomplished, such as 
the one based on Bark wavelet transform parameters, the one based on subspace analysis, and ones in daptive 
thresholding approach or using fuzzy rules, only to mention a few. As an illustration, let us take a glance at the two 
followings. 

2.1. based on Bark wavelet transform parameters 

The detection on voiced and unvoiced based on Bark wavelet Transform parameters is to make use of the 
ability of frequency segmentation and energy focusing on Bark wavelet to extract statistic parameters of speech 
signals in sub-bands. The processing flow chart can be shown in conclusion as follows(chart 1). 


Chart 1 The unvoiced speech processing flow 


Beginning Bark wavelet transform, , Extraction of Speecch signal, Calssifiers 


Framing Parameters extraction Statistic parameters 


2.2. based on subspace analysis 

The detection on voiced and unvoiced based on sub-analysis is to work out redundant information in 
speech signals. The algorithm is based on statistical analysis of the above mentioned wavelet-based frequency 
distribution of the average energy, zero-crossing rate, and average energy of short-time segments of the speech 
signal. The algorithm first classifies the input speech into voiced, unvoiced and uncertain parts by comparing 
features with predetermined thresholds. Then, the uncertain parts are treated in three conditions and dynamic 
thresholds are computed by extracted features of the input signal. Finally, the dynamic thresholds are used to 
classify the uncertain parts. The performance of the algorithm has been evaluated using a large speech database. The 
algorithm is shown to perform well in the cases of both clean and noise-degraded speech. 


IV. Experiment and result for detection 


Only facts can count. In order to get best performance for the detection on voiced and unvoiced, we tried 
our experiment in the location of language lab, gate of Southeast University and open space. 
1. Experimental subject 
Our research aims at the solution to the detection on voiced and unvoiced for programs of “Model and 
Application for Internet-based English Oral Test System”, “Teaching Model Reform under the Platform of 
Internet-based English Oral Test System” and “Model and Application Network for CET”. As the first step to 
realize auto-scoring performance, we did the experiment in the hope to solve punctuation recognition problem. 
2. Examinee 
In the first experiment, we asked 10 post-graduates(6 male, 4 female) in English class in the Foreign 
Languages Learning Center to read 210 sentences from their textbook. So in two periods, English teachers could get 
2100 sentences. According to the computer-engineers’ design, 1000 sentences were for training purpose while 1100 
sentences were for recognition data mining. The average speed of reading were designed at 8.2words/s, a bit slower 
than the speed of VOA in Special English, in a sample of frequency at 12KHZ, with a filter of window in 2133ms 
length and 10ms offset. We asked the students to do the other experiments with the same requirements, both in the 
gate and in the open space. 
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3. Condition 

The experiments subjoined natural noise so as the meet the real situation in language lab in the iB- Cet. 
Only in this way can we get the best result we were expecting before. The noise was made by the computer, 
designed for the sound/noise ratio at 20db, 10db, 5db and Odb. 

4. Procedure and result 

Our experiments applied subspace analysis to detection on voiced and unvoiced for punctuation recognition 
in programs of “Model and Application for Internet-based English Oral Test System”, “Teaching Model Reform 
under the Platform of Internet-based English Oral Test System” and “Model and Application Network for CET”. 

At the beginning, we tried to find out the shortest path for algorithm in characteristic space by using 
Nearest Feature Line(NFL in short), which is illustrated in the following chart(chart 2 and 3).b 


Chart 2 characteristic points on NFL Chart 3 the shortest path in algorithm 
- “4 RIP 


fe 6 eA 


Secondly, relying on subspace analysis, we input recorded sound hinted into the subspace characteristic of 
sound, figuring out voiced and unvoiced according to the volume of recorded sound hinted(Zhao Li, Zheng Yumin, 
& et al., 2001). 

Thirdly, we tried to figure out the related matrix based on training data, thus finding out the features and 
volume of the related matrix. To make it simple, the threshold value(” ) is the decisive vgtup ynd when the final 
value is bigger than , it means voiced, and the other way round when is smaller: 3 >@. Hereby, X 
stands for Feature vector in each frame, with i times in subspace. B stands for matrix of recdrded sound hinted. The 
experiments results are demonstrated in the following(table2 and Chart 4). 


Table 2 experiments results Chart 4 voiced and unvoiced detection 
illustration a) hb _ > o_o ee 
SNR? 20db 10db 5db Odb : a CTS ge 
Beginning 100 100 98.7 97.7 aa coors 
End 100 100 98.3 96.7 7 So 
Average 100 100 98.5 97.2 cop HLT, EPERR cee oe ne eee cc ne 


During the experiments, the detection went on in each frame, and the detection result comes from the figure 
with tolerance in 10 frames without mistakes. From table 2, we can arrive at a satisfactory result as expected with a 
average recognition of 97.2%, and in chart4, the voiced and unvoiced detection is well and clearly demonstrated. 


V. Punctuation recognition 


After we accomplished the task in detection on voiced and unvoiced, we can build a set of rules to 
disambiguate a punctuation mark in composing software to visualize punctuation marks. The rules are defined as 
follows: 

. result := T (true, punctuation mark is the unvoiced before the voiced) 
. if the preceding token is a number sound, result := F (false) 
. if the following token is with the voiced and followed by the unvoiced, result := T 
. if the preceding token is voiced in rising tone and followed by unvoiced, result := T 
. if the preceding token is voiced in falling tone and followed by unvoiced, result := T 
. return the actual value of result 
As for special punctuation, such as comma, period, exclamation or question mark, we apply the data 
described in Table 1 in matching the length in “ms”. Besides, we can apply emotional speech recognition to 
accomplish exclamatory and questionable marks in punctuation(Wang Zhiping, Zhao Li, & Zou Cairong, 2006). We 
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can also use a method to split a noun-phrase(NP in short) merging for missed punctuation so as to choose a lower 
threshold in order to reduce NP merging, resulting in more splits that shorter speech sentences. Thus, comma and 
period for example, in shorter sentences, are more likely to break entities, especially if they involve long 
phrases(Chart 5)( Benoit Favre, Ralph Grishman, & et al., 2008). 


Chart 5 comma and period inserted in threshold 
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VI. Conclusion 


The concept presented in this paper focuses on detection of voiced and unvoiced so as to accomplish 
punctuation recognition. The paper also made some touches on speech NP splitting and emotional speech 
recognition. The voiced and unvoiced detection has great to do with punctuation recognition in speech. It is very 
important and vital for the iB-CET in oral test, carrying realistic and practical value in Chinese English teaching. 

The importance of doing this research lies in the concept, which was based on paper rather than on speech 
itself. Since the notion of a sentence is very different in speech compared to written text, it is very crucial in 
education when technology has reached the peak to match human’s intelligence. 
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Abstract 


This research is proposed to improve the development of the open educational resources in the connectivist 
environment, by focusing on the creative thinking skills in Thai secondary students. The aims of this research are in 
twofold, firstly, to develop the open educational resources (OER) which promote creative thinking skills in Thai 
secondary students, and secondly, to evaluate the use of the open educational resource (OER) on the development of 
the creative thinking skills in Thai secondary students. The study was conducted with 100 secondary students from 
Chanhunbamphen school, Thailand. The students were required to study English by using an open educational 
resource (OER) system in the connectivist environment. The learning achievement test, and the creative thinking test 
were used to measure the level of learning outcome and creative thinking skills. The data were analyzed by using the 
arithmetic mean, standard deviation, and t-Test Dependent. The results indicated that the high-order thinking skills 
score of Thai secondary students significantly increased when comparing between the pre-test and post-test score (p 
= 0.01). This study suggested that the Open Educational Resources development should consider the connectivist 
environment and 5R of openness concept to promote high-order thinking. The research further suggested five phases 
of OER development, namely, 1) Stimulate 2) Scope 3) Improve 4) Integrate 5) Allotment. 


Keywords: Instructional Design, Teacher Education, Open Educational Resources, Creative Thinking skills 
1. Introduction 


Thinking skills are important. The process of thinking is complex, diverse and mixes of many skills. Teaching 
is the key to develop the higher-order thinking skills by relating to the learner’s effort and the interest in learning. 
Teaching practices must be appropriate and parallel with the socio-cultural environment which is constantly changing 
(Gonzalez, 2012). Especially, the advanced technology these days can support classroom activities including gaining 
and receiving the contents, information and data through the Open Educational Resources (OER). The OER has been 
recognised as one of the learning tools that could facilitate the acquisition of knowledge. Retrieving information from 
the internet is convenient and also without costs. The sources are various such as open courseware and content, open 
software tools, open materials, Repositories of Learning Object, websites, personal blog or even the comments in 
social media in the form of graphic and narrator which is easy to understand and stimulate the learning and creative 
thinking skills. Using these technologies for classroom activities is the important factor for teaching. Moreover, using 
the tools in the appropriated time can help the learners to achieve the learning goals as well. The OER provides virtue 
opportunity of students to obtain the knowledge in an effective and efficient way. Despite the potential benefits offered 
by using OER, there is considerable little research on how to effectively use the OER to support the learning in 
Thailand. In addition, fostering students’ thinking skills to adopt the OER is yet to be investigated in Thai education. 


2. Research Objectives 


The following research objectives were framed to guide the development of this research: 

1. To develop the open educational resource (OER) which promote creative thinking in Thai secondary 
students. 

2. To investigate the use of the open educational resource (OER) in promoting the creative thinking skills in 
Thai secondary students. 
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3. Statement of Problem 


In this research, the problem addressed by the researcher focused on the lack of using open educational 
resources to promote students’ high order thinking skills in Thai educations. After conducting the empirical 
investigation on prior research, the researcher found that the aspects related to the creating skills of thinking with the 
current learning materials administrated were mainly conducted in a traditional way. As such, classroom activities 
which related to the use of the open educational resources for Thai students are still lacking. In addition, fostering 
students’ thinking skills to learn in open educational resource context is yet to be investigated in Thai education. 


4. Connectivism Theory 


The theory of Connectivism was developed by George Siemen (2005). He mentioned that Connectivism is 
one of the educational theories that could support the rapid change of knowledge and discovery of new knowledge. In 
current digital age, the knowledge life is shorter than it used to be in twenty years earlier (Gonzalez, 2012). This is 
due to the fast development, progressively evolving of the technology which consequences shorten the time spent on 
new knowledge discovery from the immense available of information. As a result, learning solely through classroom 
based is not sufficient anymore. People need to develop themselves to handle the changing of technology and culture 
as well as the new knowledge that might impact their lives. 

Siemens (2005) defined eight principles of Connectivism, which are: 

Learning occurs through a diversity of perspectives and opinions regarding the subjects. 

Learning is constructed through the network of nodes from a particularly important source of information 
Learning is not merely a result of learners but also technology discovery 

Capacity to continuously acquire knowledge is more important than what knowledge had existed. 
Supporting and maintaining the connection is vital in supporting continuous learning. 

The ability to recognize the connectivity between the existing information, opinions, and concepts is one of 
the central skills in learning 

e To have an up to date and accurate knowledge is the main aim of Connectivism theory 

e “Decision-making is one of the learning process”; which learners need to pay particular attention by taking 

into account the reality. This is due to the knowledge as of today might be obsolete or otherwise proving to 

be wrong in the future. 
Siemens (2005) also proposed six main components required in learning based on Connectivism theory, which are: 
A platform where both instructors/teacher and students/learners could be connected 
A platform for an individual to express their ideas (blog, journal) 
A platform where everyone can discuss, share and exchange the ideas (discussion forum, open meetings) 
A platform to search and explore the existing knowledge (portal, website) 
A platform where students/learners can learn in a structured course (courses, tutorials) 

e A platform to be updated regarding the new information or knowledge (news, research) 

Connectivism drawing the concept of knowledge acquisition in the digital age where information and knowledge 
distribute and spread all over the network. Learning can occur at any places and in the informal learning environment. 
Learning is no longer an individual cognitive process but rather in a network of a community. In a network, learning 
arose from the sharing of knowledge, exchanging of ideas and opinion, communication between each node of 
knowledge by taking into account the rapid change, up to date and accuracy of information. The connection between 
each node of knowledge required different kind of supporting activities, learners and suitable time in order to create 
the most effective learning environment. 


5. Open Educational Resource (OER) 


Knowledge sharing can distribute through various approaches such as writing a blog, sharing on social media, 
uploading on YouTube. Open Educational Resource (OER) is one of the effective approaches which allow everyone 
to have free access to the educational related materials. The development of the OER system requires information 
technology management and other supporting tools. OER is accessible through public domain which defined by the 
copyrighted license indicated the permission to use the material for educational purposed. The information available 
in OER covers from learning material, course material, modules, text book, video lecture, course exercise and 
assignments as well as examination papers. To access this information, related applications and tools are required. 
Current use OER systems and applications available in public domain are included Open Courseware, Open Education 
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Resources free software, Open Source software, Open Source Initiative, Open Content, Open Publication License and 
Creative Commons. OER can be categorized into 3 main forms which are content, tools and capacity (Downes , 2009). 

e Content: in general, most Open Courseware applications can be considered as a content form which 
focused more on offering a different kind of learning materials. 

e = Tools: tools can be used to develop learning material and medium. Currently, both free and commercialized 
versions are available worldwide. 

e Capacity: referring to the EOR systems that contain a large amount of information and providing with the 
effective searching capacity. This type of OER systems is sometimes referred to as a digital library. 

Wiley (2014) identified 5 “R’s” characteristics of OER as: 

e Retain: the right to make, own, and control copies of the content. 

e Reuse: the right to use the content in a wide range of ways. 

e Revise: the right to adapt, adjust, modify, or alter the content itself. 

e Remix: the right to combine the original or revised content with other open content to create something 

new. 

e Redistribute: the right to share copies of the original content, your revisions, or your remixes with others. 

However, accessing, reusing, redistributing, revising and remixing the available materials in OER is restricted 
by the copyright license and different type of license offering a different degree of actions one can perform with the 
materials. Creative Commons (CC) (Creativecommons.org, 2002) license is one of the most frequently used licenses 
which can be divided into 6 different licenses, include: 

e CC-BY requires the user to give the proper credit to the original works with the permit to be used for 
commercialization. 

e CC-BY-SA requires the user to give the proper credit to the original works and indicate if the changes had 
been made with the permit to be used for commercialization. Redistribution of the new version of material 
under the same license. 

e CC-BY-ND requires the user to give the proper credit to the original works and indicate if the changes had 
been made. The materials are allowed to be used for commercialization. However, users are not allowed to 
redistribute the modified version of the works. 

e CC-BY-NC requires the user to give the proper credit to the original works and indicate if the changes had 
been made. However, users are not allowed to redistribute the modified version of the works. 

e CC-BY-NC requires the user to give the proper credit to the original works and indicate if the changes had 
been made. However, this material or modified version is not allowed to be used for commercial purposed. 

e CC-BY-NC-SA requires the user to give the proper credit to the original works and indicate if the changes 
had been made with the restriction not to be used for commercialization. Redistribution of the new version 
of material under the same license. 

e CC-BY-NC-ND requires the user to give the proper credit to the original works and indicate if the changes 
had been made. However, commercialization and redistribution of the modified version are not permitted. 
Open educational research is the tool to support learners in learning and gaining creative thinking skill. The 

data based in the library is also the open educational resources which include the information, tools and media. The 
data based has the specific access and publish by creative common that makes learning complete. 


6. High Order Thinking Skills (HOTs) 


Creativity skill is one of the demanded skills. The creativity is defined as "the cognitive skill of proposing a 
solution to a problem or making something useful or novel from ordinary" (Hwang, Chen, Dung, & Yang, 2007, p. 
193). It involves a wide range of higher order thinking. Andersoon and Krathwohl (2001) revised Bloom’s taxonomy 
of learning theory by considering the level of thinking based on its complexity. The taxonomy includes: 
1. Remembering is to recognize and able to recall the relevant knowledge from long-term memory. 
2. Understanding is understanding the meaning through interpretation, comparison and explanation. 
3. Applying is the implementation or use of the knowledge learnt. 
4. Analysing is to determine related to overall structure or manage different 
5. Evaluating is a decision that has been reviewed and criticized. 
6. Creating is to put the elements together to form coherent or the collaboration of all components. 
The higher order thinking involves the analysing, evaluating and synthesizing the knowledge obtained in the 
earlier stage (Andersoon and Krathwohl, 2001). To be able to obtain the creativity skills, a student needs to be able to 
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understand the content clearly, able to apply, analyse evaluate and putting the relevant knowledge together to 
generating the new ideas. 

According to Guilford proposed the concept of "divergent thinking" in the 1950s, when he noticed that 
creative people tend to exhibit this type of thinking more than others. He, thus, associated divergent thinking with 
creativity, appointing it several characteristics, as follow: (Guilford 1971 cited Aree Rangsinan 1989) 

e Fluency refers to the ability to produce a great number of ideas or problem solutions in a short period of time 
e Flexibility refers to the ability to simultaneously propose a variety of approaches to a specific problem 

e Originality refers to the ability to produce new, original ideas 

e Elaboration refers to the ability to systematize and organize the details of an idea in a head and carry it out 

According to Osborn’s method of Brainstorming, Osborn claimed that there are two principles which 
contribute to “/deative efficacy”. They are the defer judgment and reach for quantity. Following these principles, he 
established seven steps of brainstorming with the intention to reduce social inhibitions among group members, 
stimulate idea generation and increase the overall creativity of the group. These seven steps are 1.) orientation 2.) 
preparation 3.) analysis 4.) ideating 5.) incubation 6.) synthesis and 7.) evaluation. 

Creative thinking is the highest thinking skills which learners should be developed by continuously 
integrating the skill into the lessons and classroom activities. According to Dhevakulcreative (2009), thinking skills 
could help the learner to improve their capabilities, patience, intuitiveness and interests in their tasks, also making use 
of their time to improve their life. 


7. Creative Open Educational Resource (CreativeOER) 


Designing for learning is important specially to develop a higher order of thinking. Therefore, activities 
involved, learning instruction and the tools usage should be carefully designed when developing the higher order of 
thinking. Wiley (2014) suggests 5 Rs component of openness as having a significant impact on learning. Incorporating 
the 5Rs with the creativity characteristics, the Creative Open Educational Resource (CrOER) is proposed as presented 
in Figure 1. 


Connectivism (Siemens, 2005) 


The 5Rs of Creative 
Openness Thinking 
(David Method 
Wiley 2014) (Osborn, 196 

3) 


Creative Open Educational Resources (CrOER) 
High Order thinking 


FIGURE 1 Creative OER framework 
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8. Methodology 
8.1 Research Design 


This is a quantitative study that covers five steps in the research procedure. Firstly, this research began by 
analyzing and synthesizing the principles and learning theories on the related field which were Connectivism theory, 
the open educational resources and high order thinking skills. In the second step, OER was designed by using Creative 
OER framework. After that, OER was developed according to the design in step 2, so that the students could participate 
in activities provided for this study. In the fourth step, the Creative Open Educational Resources (CrOER) was 
implemented to promote high order thinking skills in the classroom. Finally, this research study was carried out to 
evaluate the CrOER, by using the pre-test and post-test. Data were analyzed by using purposive sampling and 
arithmetic mean, standard deviation, and t-Test Dependent. 


8.2 Research Procedure 


In this study, the total number of 100 secondary students from Chanhunbamphen school participated in this 
study. Figure 2 presents the research procedure used in this study. The study began by asking students to complete the 
quiz to measure the creativity skills at the beginning of the study, i.e. before students were exposed to the learning 
activities by using Google Form. The creative thinking skills assessment was developed based on Guilford theory by 
addressing four components of creativity namely, fluency, flexibility, originality and elaboration. After completing 
the pre-test, students are then introduced to the learning activity. The learning activities were designed and divided 
into six steps as follow: 

1. Search Problem: students were asked to watch a video clip introducing to the learning problem. Then, they 
were asked to analyze the problem from the media and post it on OER learning platform to brainstorm with 
their classmates. Social media such as YouTube and Facebook are used as a mean of communication. 

2. Set Up Idea: in this step, students were asked to do some research to generate the ideas to solve the identified 
problem. The teacher provided some guidance and instruction on how and where to look of reference and 
relevant knowledge. The tools used in this step are OER webtools and YouTube. 

3. Select Idea: After generating the ideas, students were guided to select and provide the supporting reason on 
how the problem could be solved by using the identified ideas. The students were then asked to present the 
potential guideline on how to solve the problem by considering the reliable references and share the 
information with other classmates. Teacher and other classmates would be asked to provide comments and 
raise any questions and issues that might have an impact on the given problem and the presented solution. 
Several tools could be used as this steps such as OER Websites, Youtube ect. 

4. Summary Idea: In this step, students were asked to summarize the guideline to solve the problem as well as 
a suggested solution by synthesizing the information and comments given by their peers and teacher to 
generate the new knowledge. The OER tools used in this step were Facebook to communicate the idea and 
the OER websites. 

5. Share Idea: After summarizing the idea, students were required to present the new knowledge gained in step 
4. The OER tools and elements that could be used in the presentation are Video Clip, e-book, Web site, Page 
Facebook or Blog. 

6. Score Idea: both teacher and students were asked to evaluate the idea presented in step 5. The teacher 
evaluated the assignment according to the criteria and providing the comment. Students were asked to provide 
feedback on the presented ideas of their peers. The OER tools used in this step were Facebook, Youtube. 

After complete the learning activities, students were asked to answer to the creativity test questions. 
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Pre-test 


Search Problem 


Set up Idea 


Select Idea 


Summary Idea 


Share Idea 


Score Idea 


Post-test 


FIGURE 2. 6S of Creative OER Procedure 


9. Analysis and Discussion 


The question items of pre-test and post-test to evaluate the creativity of students were evaluated according 
to the corresponding creativity components and the level of creativity as presented in Table 1. 


Table 1: The criteria of creativity measurement 


Components Level of creativity 

of Creativity Good Fairly Need for Improvement 

Fluency The student was able to The student was able to The student was able to 
correctly answers more correctly answers at least —_ correctly answers less 
than 70% of the question 50% but not more than than 50% of the question 
items 70% of the question items items 

Flexibility The student was able to The student was able to The student was able to 
organise, categorise most organise, categorise the organise, categorise some 
of the items in variety items in three to five of the items in one or two 
approaches number of approaches approaches 

Originality The student was able to The student was able to The student was able to 
adjust, alter, modify, and adjust, alter, modify, and —_ adjust, alter, modify, and 
apply most of the items apply some of the items apply a few of the items 
accordingly accordingly accordingly 

Elaboration The student was able to The student was able to The student was able to 


elaborate, systematize and 
organize the details of most 
of the items accordingly 


elaborate, systematize and 
organize the details of 
some items accordingly 


elaborate, systematize and 
organize the details of a 
few items accordingly 
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Table 2 presents the frequency and percentage of students’ creativity score obtained from the 
questionnaires. The score highlights the pre and post test score according to four dimensions, namely, fluency, 
flexibility, originality and elaboration. 


Table 2. Frequency and percentage of students’ creativity before attending the class 


Components Level of Creativity number (%) 
of Creativity 
Good Fairly Need improvement 
before after before after before after 
Fluency 21.4 57.1 37.1 42.9 41.5 0 
Flexibility 25 46.4 39.3 42.9 35.7 10.7 
Originality 17.9 42.9 32.1 50.0 50.0 7.1 
Elaboration 10.7 32.1 17.9 46.4 71.4 21.4 


In this study, the Creative OER Procedure which consisted of six steps, 1.) search problem 2.) set up idea 3.) 
select idea 4.) summary idea 5.) share idea 6.) score idea (6 S CrOER) to promote creative thinking was implemented. 
The level of creativity was divided into 3 levels which are good, fairly and need for improvement. The good creativity 
level shows to have a higher proportion of items based on the post-test score in every creativity component as 
compared to the pre-test. Similary, the fair level of creativity also had an increasing percentage of post-test scores in 
all components. In contrast with the first two levels, the number of items that were categorized as needing further 
improvement level of creativity showed significantly lower post-test score in all components. This result indicates the 
improvement in fluency, flexibility, originality and elaboration of the students after experienced with the OER 
application. The items that were initially categorized as the need for improvement of the fluency has reduced to zero. 
This indicates the potential usefulness of OER to improve the creativity level of the secondary students. Finally, based 
on the evaluation, this tool has been evaluated at “The Most Satisfied Level” by both teachers (instructors) and students 
(attendees). 


Table 3. The pre-post test score (T-test) 


Test n Points S.D. t sig. 
Pre - Test 100 12.41 2.34 9.967 000 
Post — Test 100 16.76 1.43 


Based on the initial result, the statistics were used to observed the significant level of the pre and post-test. 
Thai secondary students’ posttest score gained from high order thinking skills part was higher than the pre-post test 
score at the statistical significance at level .01. 


9. Conclusion 


This research presents the use of OER as a tool to promote the creativity skills for the Thai secondary students. 
The study consisted of 100 students. The designed learning activity involved six steps, which are, problem searching, 
set up the idea, selection of the ideas, summary, share and score the selected idea. The OER was used as a learning 
platform to carry out the activity. The OER tools used in this study consisted of several tools which could be divided 
into two categories. The tools used to support learning were E-paper, search engine, YouTube, and the other uploaded 
online materials/Resource and the tools which used for communication such as a forum, blog, news as well as social 
media such as Facebook. The research study was carefully addressed the Creative Common licensing agreement. The 
pre and post-test were used to examine the level of creativity. The result indicated the higher level of creativity after 
students were experienced with the OER based learning activity. This result suggested that the OER has the potential 
to promote the higher order of thinking and creativity skills. 
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Abstract 


In this study the Turkish Language MOOC: “Tiirkce OSreniyorum” is presented. Firstly, the language MOOC 
phenomenon is introduced, and the trend and issues about language MOOCs are addressed briefly. Secondly, the 
case of “Tiirk¢e OSreniyorum” will be shared with related statistical research data of 6254 participants. Thirdly and 
finally, conclusion and a few final remarks on the foreseeable research directions within the field of MOOLCs is 
shared. 


Keywords: MOOC, MOOLCs, Massive Open Online Courses, Tiirkce OSreniyorum, Learner Profile 
Introduction 


Large-scale teaching traces back to first implementations of distance education in the 18th century. It takes 
its roots from teaching machines, computer-assisted instruction, learning management systems, open education and 
open educational resources movement and continues to evolve with the emergence of massive open online courses 
(MOOCs) (Sanchez-Gordon & Luja‘n-Mora, 2018). MOOCs are freely accessible learning materials and media 
used for learning, teaching and assessment (Barcena & Martin-monje, 2015). The term MASSIVE implies that a 
MOOC should allow the access to a very large number of students such as from 1,000 to 100,000 students, without 
leading to a significant problem for its functioning (Sanchez-Gordon & Lujan-Mora, 2014). The term OPEN has 
several meanings such as open enrolment, open content, open learning platform and being free of charge. Open 
enrolment means that course should be open to students outside the university that organizes it, and it should not 
require any prerequisites, such as previous studies or a degree possession. Open content implies that the content 
generated by the course should also be published with an open license so that the new content could be reused by 
others. Open learning platform indicates that the course contents are spread in different places such as websites, 
blogs, wikis or multimedia repositories. In addition, an open learning platform can also mean that the technology or 
the platform on which the course is offered is also open. Finally, the course should be free of charge. The term 
ONLINE implies that the course is done remotely via the Internet, and it does not require physical attendance at a 
classroom (Gaebel, 2013). 


The first MOOC - Connectivism and Connective Knowledge organized by George Siemens and Stephen 
Downes- was offered in 2008 for free. Approximately 2,300 students enrolled in that course. A big increase 
occurred in the number of both courses and students in a decade. According to the report of MOOC search engine 
Class Central (2017), over 7.000 of MOOCs delivered by single university platforms and 33 big MOOC providers 
from all over the world as of 2017. The number of students is over 50 million (Sanchez-Gordon & Luja’n-Mora, 
2018) as of June 2017, Table 1. 
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Table 1. Statistics for Main MOOC Providers as of June 2017. 
Provider Country Students(millions) Courses Partners 
Coursera United States 25+ 2,000+ 150+ 
edX United States 10+ 1,500+ 100+ 
XuetangX China tiga 400+ N/A 
Futurelearn UK 6+ 400+ 100+ 
Udacity United States 4+ 170+ N/A 
MiriadaX Spain 3+ 600+ 100+ 
FUN France 1+ 270+ 90+ 
MexicoX Mexico 1+ 60+ 40+ 


Resource: Sanchez-Gordon & Luja’n-Mora, 2018 


MOOLCs: Massive Open Online Language Courses 


In the current context of a globalized and technologically connected but still linguistically diverse world, 
the knowledge of one or several languages is an indispensable skill to live, work or connect technologically and 
socially (Perifanou, 2015; Perifanou and Economides, 2014). It also must be combined with digital ones in the 
contexts of work, personal and social life at the local, regional and global levels (Pegrum, 2016). Open access to 
language courses and resources is needed more than ever before in order to meet the needs of linguistically and 
culturally diverse learners who live and work in a global society (Perifanou, 2015). This justifies requires to create, 
measure and evaluate second language teaching initiatives in new formats such as MOOC since they have a great 
potential to present a genuine, free and collaborative learning environment which is open to different geographical, 
temporal or social situations (Cuevas, 2017). Nowadays, the number of MOOCs and Open Educational Resources 
(OERs) connected to foreign language learning have increased, and there is a growing interest for them. However, it 
is scarce in comparison with other disciplines such as economics, technology or science. While Perifanou (2016) 
reported 29 MOOLCs from 67 different MOOC platforms/providers in total as of 2015, MOOC search engine Class 
Central reports 191 foreign language MOOCs with 106.5k followers as of October 2018 (Class central, 2018). In 
addition to providers listed in Class Central Mixxer, Instreamia, OpenLearning and TandemMOOC platforms 
provide MOOLCs with a connectivist way of learning and by emphasizing the role of social and cultural context 
(Perifanou, 2016). 


Although a considerable amount of research has been conducted in the last decades with regard to digitally 
enhanced language teaching and learning that has shown the tremendous possibilities technology can offer 
(Perifanou & Economides, 2014), there is a limited research on MOOLCs since it is an emerging technological 
platform and it is still considered as "neonatal" (Sokolik, 2014). From a methodological point of view, MOOLCs 
appear to have great potential for self-directed language learning as: (a) there is a possibility for the use of 
audiovisual materials; (b) the use of the Internet allows oral and written interaction between language learners at the 
same proficiency levels; (¢) MOOLCs favor collaborative learning; (d) allow language learners to proceed at their 
own pace,(e) they promote learner autonomy; (f) encourage the sharing of Internet-based resources; (g) favor 
empathy and cooperation between students sharing the same interests. (Chacon-Beltran, 2017). On the other hand, in 
reality, the professor who designs a MOOLC has a restricted freedom when creating the communication and 
evaluation activities and tools (Read, 2014), since it does not you can always choose the platform in which you will 
implement your course or the type of resources and activities that this platform allows you to implement. 


Linguistic communication should be the backbone of these courses (Sokolik, 2014) as second language 
acquisition theory indicates learners not only needs comprehensible input (Krashen, 1985) (i.e. activities to practice 
their reading and listening skills) but also opportunities for output (Swain, 1995) (i.e. activities to practice their oral 
and discourse skills). However, many of the existing MOOLCs have continued the tradition of online language 
courses that focused on grammar, written writing or vocabulary, avoiding the complications of oral practice in a 
foreign language and the interaction between the participants (Martin-Monje, Barcena & Read, 2014). Input 
activities can be provided easily in MOOLCs but output ones can not basically due to unbalanced teacher-student 
ratio. One of the distinctive features of MOOLCs is unbalanced teacher-student ratio unlike a traditional distance 
language learning platforms with a balanced teacher-student ratio and a wide variety of communication, feedback 
and assessment tools (synchronous & asynchronous). Most of the existing MOOLCs are essentially self-paced 
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language learning materials with low interactivity and feedback. On the other hand, many of the design proposals for 
MOOLC tend towards a new model of personalized (Zapata-Ros, 2013) or adaptive (Godwin-Jones, 2014) learning 
since adaptation to the profile and the individual interests of the participants causes better involvement in the course, 
which requires the development of more sophisticated tools by educators, designers and programmers. 


The Turkish Language MOOC: Tiirkce Ogreniyorum (Learn Turkish) 


There are two traditional distance education programs to teach Turkish as a foreign language, and a 
comprehensive, well-structured, non-profit MOOLC. Distance education programs are provided by Anadolu 
University and Yunus Emre Institute, where the MOOLC is provided by Yasar University under the name of 
"Tiirkce OSreniyorum (I learn Turkish)". 


The objective of "Tiirk¢e OSreniyorum" Project is to develop a worldwide distance learning platform to 
teach Turkish as a foreign language, which is sensitive to individual differences and can differentiate the content 
according to the needs of the learner. The primary goal of "Tiirkcee OSreniyorum" is to provide continuous global 
access to qualified self-paced learning materials for the Turkish language learning as well as to make them 
accessible to people at risk of social or professional exclusion such as irregular immigrants and refugees. It was 
developed under "the Differentiated Distance Education of Turkish as a Foreign Language" project, supported by 
TUBITAK with the project code of 115K270. It is limited by level Al, which is referenced in the Common 
European Framework of Reference for Languages 


There are 53 chapters in "Tiirkce OSreniyorum". Each chapter bearing 5-minute lecture videos, 2-minute 
real-life drama videos, reading-listening-writing-pronunciation activities, approximately 40 games. The course 
contains forum, virtual meeting tool, one Al level dictionary with 1.500 vocabulary sand one glossary for grammar 
tules. There are approximately 6000 questions to practice in total in the course. Approximately 40.000 words of text 
in Turkish translated into English, Arabic, Russian and French language to offer support to learners in the 
development process. "Tiirkce OSreniyorum" runs on Sakai LMS, which is integrated with Kaltura Video Platform 
and Google Analytics. Learning analytics data is gathered from the Diagnostics tool, Motivational Beliefs and Self- 
Regulated Learning Strategies Questionnaire (MLSQ), Logs of exercises, LMS Statistics, and Google Analytics. 
Diagnostics tool covers demographic information of learners, their primary and secondary languages, prior 
knowledge of Turkish, and home country. The aim of this tool is to collect data about learner profile in terms of the 
region they live in and the language they speak. MLSQ is used to detect cognitive and affective readiness of the 
learner in terms of intrinsic value, test anxiety, cognitive strategy use, and self-regulation (Pintrich & DeGroot, 
1990). Logs of exercises record activity events of the learner (number of tries, given responses etc). Sakai CLE 
collects site usage statistics regarding learner visits, tool activity, and resource activity. Google analytics provides a 
rich source of data such as learner preferences (demographics, interest, age, geo, etc.), time spent on modules, page 
statistics (page view, page exits, behavior flow, etc.), preferred technology (device, browser, operating system, 
screen resolution, screen colors, etc.), video events (video view duration, numbers of play, pause and skip attempts, 
etc.). 


Project team contains 2 instructional designers, 4 subject matter experts, 1 graphic designer, 1 video 
developer, 1 multimedia developer, | photographer, 1 system administrator, 1 software developer, 4 translators, 7 
actors, 6 voice actors, 3 cameramen, 2 production specialists, 2 post-production specialists, 1 cinema director, 1 
scriptwriter, 1 assessment and evaluation specialist and 2 consultants. 


Structure of content of "Tiirkge OSreniyorum" was based on five theoretical approaches (Ozan, et al., 
2018). First one is the theory of transactional distance, which establishes a relationship between dialog, structure, 
and learner autonomy (Moore, 2007). Second one is theory of self-regulatory learning, which is critical especially in 
distance education because of the absence of the very active role of the instructor and the requirement for high- 
autonomy (Hsu et al., 2009), goal setting, self-monitoring, self-evaluation, use of task strategies time planning and 
management (Dabbagh & Kitsantas, 2005). The third one is the Theory of Multimedia Learning (ToML). Forth one 
is Tomlinson's (2001) approach for differentiation of instruction. The final one is Notional-Functional Approach 
(Ellis, 2005; Halliday, 1986; Hymes, 1971) as second language teaching theory. Learners can follow the content 
according to his/her interest, pace and needs. They can follow the content in any order, they don’t need to follow it 
in a linear sequence. There is no time limitation to complete the course. Self-assessment tests and feedback in each 
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activity were placed in each module. Learners can take those self-assessment tests multiple times, there is no 
limitation in the number of trials in tests as well. 


6254 learners from 136 different nationalities and 104 different countries enrolled in the course as of June 
2018. 31.68% of them were female and 65.80%of them were male. 158 (2.52%) people did not want to indicate 
their gender. 26,16% of those who want to learn Turkish as a foreign language is between 18-24, 40,87% are 
between 25-34 years of age, and rest of them (28,61%) is above 34. 54.28% of learners learn Turkish because of 
academic purposes. 44,42% of them listed business as a reason. 


50,98% of the learners are Syrian.However, only 2.13% of the them stated that they were lived in Syria. 
The majority of the registered learners (66.49%) live in Turkey. The mother tongue of 81% of learners is Arabic. 
Arabic is followed by English (2.94%) and Persian (1.58%) respectively. 60.70% (n = 3796) of the learners do not 
know any other foreign language. 32,08% (n = 2006) currently speaks one foreign language. 


11% of the learners have primary or secondary education, 31% of them have high school or equivalent 
degree, 44% of them bachelor degree, 11% of them have a masters degree and 3% have Ph.D. degree, 


A chi-square test of independence was performed to examine the relation between residency and education 
level. The relation between these variables was significant, y2 (3) = 237,104, p < .001, Cramer’s V = ,20. Learners, 
who live in Turkey, less likely have a graduate degree. 


44% of learners work. 29% of them are students. 15% of them stated that they were not working. 74% of 
them do not have an online course experience. 68% of them uses social media. Men are more interested in 
"Business", "Developments in the World", "Science and Technology" and "Sports" than women while women are 
more interested in "Fun and Life", "Language and Culture" and "Literature" than men. 


Conclusion 


In this study, language MOOCs discussed briefly and the Turkish Language MOOC: “Tiirkce 
Ogreniyorum” was presented a case. A satisfactory model for designing language MOOCs has not been found, yet. 
It is an emerging field. However, there is a consensus on that courses must be accessible in different devices, 
including mobile devices, and be integrated into the real life of the participants, taking into account the different 
special learning needs that may arise and the people at risk of exclusion (Morgado, Teixeira and Jansen, 2015). 
Integration of social networks and peer review activities into this large-scale teaching platforms, which has an 
unbalanced teacher-learner ratio, might transform participants from individual learners to social agents who have 
communicative competence. Self-regulation is also critical especially in MOOCs because of the absence of the role 
of the instructor and the requirement for high-autonomy. Learners in MOOCs are responsible for their own learning 
by arranging the time, pace and strategies. “Tiirkce 6greniyorum” has low learner-instructor dialog. Therefore, it 
was designed in a less structured context to reduce transactional distance. 
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Interactive Video (IV) is receiving increased attention in higher education in recent years. The concept of 
IV shifted with the development of the IV technologies, especially with the further development of the assessment 
and learning analytics features. This paper briefly reviews the research about IV use in higher education. It also 
introduces our experience in using one of the IV tools at Purdue University. Future research is suggested based on 
our research and experience. 


Development of IV 


IV has been used in higher education for several decades. Combining tutorials and simulations with 
reflective questions, the early IV was a supplemental learning resource that helped students to understand the 
abstract textbook content (Hansen, 1989). With the student self-paced learning design, early IV not only provided a 
medium for observation but also provided an environment for learning reflection (Hansen, 1990). However, the 
early IV is considered similar to computer-assisted instruction (CAI) (Hansen, 1989), which is effective in designing 
drill and practice of basic skills (Kulik, 1994) but failed to adapt technology to fit the needs of multiple learning 
preferences (Mayer, 2009). 

With regard to the limitation of the early IV, Hansen (1989) pointed out the future IV design and 
development should include the following principles: first, it should follow a student-centered approach; second, it 
should provide multiple interaction approaches; third, it should serve as a model for student self-assessment; fourth, 
it should provide feedback to students; fifth, it should be able to monitor student progress and provide the instructor 
with a profile of the students’ understanding; and sixth, the design should be open and allow for future revision. 

Currently, the most popular IV tools in higher education are Kaltura, Camtasia, Adobe Captivate, 
Articulate 360, Edpuzzle, and PlayPosit (Hayden, Fleischer, & Taylor, 2017; Kolas, 2015). These tools are no longer 
simply a combination of video and questions. These tools intend to ensure videos are being viewed and encourage 
student engagement, improve retention of knowledge, and promote deeper reflective learning (Hayden et al., 2017). 
In addition, they provide multiple features to meet diverse teaching and learning needs, such as video editing, 
embedding questions, and providing feedback. These tools can incorporate text, images, external links, maps, 
multiple-types of quiz questions, polls, and discussions directly within the video. Some tools even include an open- 
source platform to encourage resource sharing between institutions. To some extent, the development of current IV 
tools is a good response to Hansen’s (1989) principles. 


The Effectiveness of ITV in Higher Education 
With the development of IV technology, researchers report that students in both e-learning as well as 
traditional classroom environments achieve better learning performance and have a higher learner satisfaction when 


IV is incorporated into the learning experience (Zhang, Zhou, Briggs, & Nunamaker, 2005). Compared to using 
traditional video in class, other research found IV helps to engage students in online class and decrease the dropout 
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rates when students watch the video (Kovacs, 2016). In addition, faculty believe that the new IV features contribute 
to increase students’ awareness when watching a video, which makes it a good tool for both formative and 
summative assessment (Kolas, 2015; Kovacs, 2016). 

Based on recent research, one of the most frequently used features in today’s IV tools is in-video quizzing. 
Research demonstrates that using testing as a study tool improves long-term retention of learning materials 
(Roediger, & Butler, 2011). In addition, quizzing and immediate feedback benefits information retention in multiple 
learning settings (McDaniel, & Fisher, 1991; Lacher, & Lewis, 2015). Griswold, Overson, and Benassi (2017) 
conducted an interesting study in a graduate occupational therapy class. Without using any IV tools, the participants 
took online quizzes through a learning management system while they were watching the pre-recorded lecture video 
on another platform. They were asked to pause the video several times to take the quizzes and were allowed to go 
back to watch the video while they were taking the quizzes. The results show the online quizzes helped to improve 
student test scores in a final unit exam. 

Today, many IV tools make taking quizzes much easier than this. Multiple types of quiz questions along 
with other interactions can be embedded into the video directly. In other words, videos and quizzes can be easily 
included into the same platform, without the need for students to access multiple areas in order to complete the 
activity. For example, Students who need additional time to work on the quiz question have the option to pause the 
video and review it. Students who are comfortable with the material and prefer to move quickly through it can do so 
as well. Captions are generally included with IV, so students who prefer to read text rather than view it have this 
option as well. Review features are especially important for students whose primary language is not the same as the 
material presented as well as students with learning disabilities. 

In Cummins, Beresford, and Rice’s (2015) study, an IV tool was used to design quizzes based on a series 
of instructional videos for a college level programming course. The researchers found the majority of student think 
the in-video quizzes were useful in helping them understand the video. The study also found that students interact 
with the videos in very different ways because of different learning styles. The researchers recommend considering 
student behavior and motivations when designing quiz questions using IV tools (Cummins et al., 2015). Figure | 
shows a screenshot of an in-video quizzing designed for an ESL program in a community college. 


= + x 


INTERACTION(S) 


Did the presenter include any of the 
following skills ? Check all that apply. 


She introduced herself 


CU) lad 


Watch later Share 
= 


Figure 1. A Check All question created with PlayPosit. 
Note: Captions are not included in this screenshot because it is the end of a student’s presentation. 


Another very popular feature in IV tools is learning analytics, which helps instructors understand and 
optimize learning and the learning environment by measuring, collecting, analyzing, and reporting data about 
learners and their context (Long & Siemens, 2011). Most of the aforementioned IV tools provide analytics on both 
course-level and individual-level. Course-level analytics show how the class is progressing with each of the 
activities and each of the quiz questions (Blackstock, Edel-Malizia, Bittner, & Smithwick, 2017). Faculty can use 
analytics to determine what to cover during class time for ensuring the specific needs of each class are met. 
Individual-level analytics show student understanding, level of effort, and other related learning behaviors on each 
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of the activities and quiz questions (Hayden, et. al 2017). This information can be used to identify struggling 
students for additional support before the student becomes too far behind. 

However research also shows the limitation of some IV analytics. A study conducted in a postsecondary 
level medical course shows the standard generic and direct feedback responses that some IV analytics provide do not 
meet students’ multiple learning style (Ovalle, Schofield, O’Hara-Leslie, & McLain, 2017). To effectively improve 
learning, students who interpret the quiz questions differently need a specific explanation for why their answer is 
correct or incorrect (Ovalle, et al., 2017). 

The current study on IV analytics is still limited. More research about student/faculty perceptions as well as 
the effectiveness and usefulness of IV analytics is needed in the future. Figure 2 and Figure 3 show PlayPosit’s 
learning analytics at course-level and individual-level, respectively. 
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Figure 2. Course-level Learning Analytics 
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Figure 3. Individual-level Learning Analytics 


Although a number of studies claimed IV tools effectively improved learning outcomes (Hayden et al., 
2017; Kolas, 2015; Kovas, 2016; Zhang, et al., 2005), there still some studies show no difference between using IV 
tools and using other teaching strategies on student test grades (Bennett, 2018; Lacher, & Lewis, 2015). Researchers 
point out students grades are not sufficient in assessing the effectiveness of IV in education (Austin, Lawson, & 
Holder, 2007; Lacher, et. al., 2015). Additional factors, such as the nature of the instructional content, instructional 
methods, and related instructional material should also be considered (Hattie, 2009). In addition, researchers suggest 
future studies focus on comparing perceived learning styles and performance on testing (Bennett, 2018), evaluating 
and analyzing a wide range of activities rather than focusing on a single assessment (Kolas, 2015), and investigating 
how students engage with the IV materials (Cummins et al., 2015). 
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Using IV Tools at Purdue University 


Purdue University has been using instructional videos in different learning environments, including face-to- 
face, fully online, and hybrid courses. Most of these videos are pre-recorded lectures/instructions, which are used as 
required learning tasks for preview and overview of specific topics. Many faculty want to use IV tools to make these 
traditional videos more engaging to all types of students. Our needs assessment shows we have more than 30,000 
full time undergraduate students and 9,000 graduate students, including about 9,000 international students from 122 
countries. A total of 7,085 students participate in some form of distance or online learning. In addition, over 100 
faculty are from non-English speaking countries. With regard to this situation and the aforementioned instructional 
video use at Purdue, we decided that the IV tool that we choose needs to include the following features: 
allow closed captioning; 
allow embedding instructions, comments, interactive communications, and assessments; 
force watching without skipping any part of the video; 
include assessment analytics; 
allow integration with Blackboard Grade Center. 


Prior to PlayPosit, Purdue had been using Kaltura as a video production tool. As mentioned before, it is 
also a commonly used IV tool in higher education. However, the limited interaction feature in this tool cannot meet 
the teaching and learning needs at Purdue. Based on our market analysis, we chose to pilot PlayPosit. Compared to 
Kaltura, PlayPosit provides more detailed learning analytics and multiple interactive activities, which helps faculty 
create instructional materials to meet diverse learning needs in both online and face-to-face class. In addition, it can 
be used as a class engagement tool that can collate real-time student responses in face-to-face classes. Table 1 shows 
a brief analysis of these two tools. 


Table 1. 
Comparison of three different IV tools used at Purdue University 


Features PlayPosit Kaltura 

Video Editing Yes Yes 

Authoring Interface Users need basic training Users need basic training 

Search Public Video Yes No 

Upload Video Yes but has limitation on size Yes 

Caption Yes Yes 

Integrate with Blackboard Yes Yes 

Types of Interactions Multiple choice, Y/N, free answer, poll, Multiple choice only 
discussion, reflection, etc. 

Course Copy Need to re-assign the link Easy to copy 

Learning Analytics Detailed analytics for individual students and Very limited analytics 
course level 

Accessibility Passed accessibility check Passed accessibility check 

In Class Engagement Yes No 


Although many faculty showed interest in this new tool, we had difficulty finding users in the first 
semester. We hosted two campus-wide showcase events in order to provide information about the tool and identify 
potential faculty interest. Research shows the main barriers to video use in higher education are course development 
time and professional development in the technology (Leahy, 2015). So we provided faculty ample time to consider 
how they might incorporate IV in their courses, and we provided professional development through technology 
training and instructions on facilitating course design with this tool. With the help of our pilot team, some faculty 
created a few sample videos for their summer courses. Most of these faculty decided to create course-wide activities 
and assessments with this tool for the following semester. Through direct assistance with the process, we found 
faculty use this tool differently in their classes. The learning materials that are created using this tool can be 
concluded in the following categories: 

e for pre-view and review 
e for online class instruction 
e for observation and reflection 
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e = for formative assessment 
e = for taking attendance 
e for lab tutorials 


We have several successful use cases during the two-year pilot. For example, in a college level planetary 
science program, the instructor initially used one interactive video in a course that he had previously taught. He used 
this as a way to assess effectiveness as compared with his previous teaching method for that particular content. This 
trial IV was used to provide an interactive online resource students could access on Blackboard as part of a face-to- 
face course. The instructor discovered that IV provided a good way for him to provide low-stakes, formative 
assessments for his students. This was important to him because his course is based on mastery of the material and 
his teaching philosophy involves allowing students to have multiple opportunities to demonstrate mastery. He set his 
IV to allow students to retake the embedded quizzes as many times as they liked. He is currently using IV as a major 
component of his totally online version of this course. Each interactive video is only worth a small percent of the 
grade, but it is one of many ways for students to perform a self-assessment of the material as well as to demonstrate 
mastery. 

During our pilot, most feedback from faculty focused on questions about IV material design and 
development. However, we also found that several technical barriers prevented some faculty from adopting this tool 
in their classes. These included: 1), limitations on size and video type that can be uploaded within this tool; 2) video 
links breaking during course copy; and 3) limitations in allowing multiple users, such as instructor and teaching 
assistant, to create/edit a same video. These problems increased instructors’ workload when they were trying to 
adopt this tool in their classes, especially when they were trying to create a large number of IV for a single course. 
Since only a few faculty created formative assessments with this tool, there was no issue reported in learning 
analytics. 

We have documented several lessons learned in order to decrease barriers to implementation. We suggest 
the following: 1) faculty limit video length to limit file size; 2) institutions use the most current LMS releases in 
order to avoid system bugs that can create issues like links breaking; 3) IV developers provide permissions or roles 
that allow faculty and teaching assistants to work together on common video assets without compromising security 
of video assets for other courses. We also recommend that faculty begin with creating a trial interactive video before 
revising an entire course using IV. This gives the instructor an understanding of the time and effort it takes to create 
an interactive video. It also provides an opportunity to discover the most effective ways to incorporate various IV 
features within a particular field. Once an instructor has discovered what works and does not for their specific 
course, additional IV can be created, minimizing revisions and wasted time. 


Conclusion 


Based on our research and experience, a good IV tool needs to be easy to use and provide multiple 
interactive activities to meet multiple learning needs. It should be seamlessly integrated with a learning management 
system, provide dynamic feedback to students, and be able to provide detailed individual/course level learning 
analytics. Researchers point out that taking advantage of the multiple forms of interaction and communication 
should be considered as a goal for IV future development (Ovalle et al., 2017). Therefore, more research studies are 
needed in evaluating different types of IV communications and activities (Kolas, 2015) and how students engage 
with these IV materials (Cummins et al., 2015). In addition, multiple factors, such as instructional content and 
instructional methods, need to be considered in the future studies (Hattie, 2009). To get the best use of IV tools, 
more research about faculty perception as well as the effectiveness and usefulness of IV analytics is also needed in 
the future. 
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Abstract 
Based on the fourth generation of time management concept, as well as the comprehensive analysis of 
common time management software, the current situation of Chinese college students’ time management and the 
psychological characteristics of Chinese college students, this research designed and developed a time 
self-management system suitable for college students, trying to provide technical support for their time 
management, help college students improve their time management efficiency and enhance their quality of life. 


Keywords—Time management; College students; System design 
I. INTRODUCTION 


College students are the hope of the nation and shoulder the mission of building the country. College 
students’ time in school is precious, and its time utilization is directly related to the quality of the country's future 
talents. However, the current time management situation of college students is not optimistic. Zhang Xiaofang has 
found that 73.3 percent of college students think that there are more problems in the use of time. 51.5 percent of 
college students are not good at planning their daily and weekly schedules according to their actual situation. More 
than 1/2 college students arrange and use their leisure time according to their feelings and interests, and less than 1/5 
students plan their leisure time[1]. There are many problems in college students' time management, which need to be 
solved urgently. 

The situation of time management of college students is poor. In many cases, it is not the subjective cause 
of college students, but the lack of corresponding concept of time management. In the face of sudden changes in 
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school life, there is no guidance, no awareness of time management at the time of admission. On the one hand, 
college students have heavy academic tasks and many things to accomplish. On the other hand, many college 
students have poor management of time, fail to make good use of time, but do not know what to do with time. Some 
students can't catch the key transaction, don't know how to arrange the order of things, in the face of seemingly 
complex transaction do not know how to start. College students tend to rely on their minds to remember their plans 
without the right tools, making them unsustainable. From the research at home and abroad, in the students' learning, 
especially the college students’ study and life, learning time management can play very important role on improving 
the students' study and work efficiency. It is also help to improve the students’ academic performance, alleviate 
psychological pressure and reduce anxiety. 

With the popularization of computer and other intelligent devices in college students, more and more 
college students use electronic means to assist their study and life, and help themselves to manage their time by 
computer. However, there are many problems with time management software for college students. 

The goal of this study is: based on the latest time management concept, design and development for college 
students’ time management system, guide students to grasp the macro time, scientific planning, and promote good 
habits. 


I. TIME MANAGEMENT THEORY AND SUPPORT TECHNOLOGY 


A. Time Management Theory 
1) Time Management Concept 

The concept of time management originates from the academic field, but time itself comes from life, and 
time management is closely related to life, so it gradually becomes a popular concept in the process of later 
development. Although the passage of time does not change the speed of its passage by any external force , it does 
not mean that we are going to be a slave to time , and on the contrary , we must manage the time through a scientific 
approach and conduct more transactions within a given time according to its own arrangement. The use of time 
should change from passive to active, systematic, focused and planned active allocation of time. The central 
principle of time management is to try to concentrate the necessary bulk time on the most important tasks. 

In the literature on "time management", many scholars believe that the true connotation of time 
management lies in people's self-management. Self-management is not simply the assignment of work or study tasks, 
but the initiative to plan, allocate and use time according to goals, guided by system theory. In the end, in addition to 
achieving the set goals, we should improve the work efficiency, explore the potential of the subject, and help the 
subject achieve a sense of achievement of self-worth[2]. The ultimate goal of studying time management is not only 
to manage time in an efficient way, but also to seek creative development of people and promote better 
self-planning. 


2) Fourth Generation Time Management 

So far, the time management concept has undergone several generations of evolution. Among them, the 
first three generations marked by “Memorandum”, ”Calendar” and “Plan Manual” respectively pursues the 
management of time too much, but neglects students’ psychological experience, which easily leads to the students 
being in a tense state for a long time. 

In contrast, the fourth generation no longer emphasizes how to arrange time and affairs, but take “important 
things first” as the first principle. “Important but not urgent” affairs can achieve a balance between output and 
capacity, focusing on it will help to improve the efficiency of time management and quality of life for college 
students. Although the fifth generation time management concept has also emerged, its main target group are 
corporate employees, which are not in the scope of this research. 

“Important things first” is represented by a four-quadrant time management matrix, as shown in Table 1. In 
this table, urgency means that it must be dealt with immediately. Importance is related to goals, and whatever is 
valuable and conducive to the achievement of personal goals is important. The first is urgent and important and 
requires immediate attention. The second type of business is important but not urgent, including networking, writing 
mission statements, planning long-term goals, and nip in the bud. The third type of business is urgent but not 
important, including sudden visits, some meetings, and some activities. The fourth is neither urgent nor important. 
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TABLE 1. |. FOUR-QUADRANT TIME MANAGEMENT MATRIX 
Urgent Don't urgent 

The second quadrant 
The preparatory work 
Preventive measures 
Projects with deadlines looming To build a relationship 
Tasks to be completed within a limited Identify new development 
time opportunities 


The first quadrant 
Crisis event 


Important 


@ Urgent problem @ Make plans and take necessary leisure 
time 
The third quadrant The fourth quadrant 
@ Some emails, some phone calls @ = Trifles, work that kills time 
Pi @ Some reports, some meetings @ Some calls and emails 
Unimportant ate oe : 
@ = Receiving visitors and public events @ boredom 
@ =A lot of immediate, urgent things @ "Escape" behavior 
@ Many popular activities @® Watch too much TV 


Generally, people are sensitive to the degree of emergency, but put off important things. Focusing on the 
first kind of business will lead to a larger and larger scope of processing, which will eventually lead to crisis 
management and occupy all the time and energy. Emergencies are usually a priority for others, but not necessarily 
for themselves. Highly effective people strive to reduce the number of third and fourth categories of business, 
whether urgent or not. Increasing the time spent on transactions in the second quadrant usually reduces the number 
of transactions in the first quadrant. 

The emphasis of four-quadrant management is on the second class of transactions. Because the second type 
of business is the balance between output and capacity. As we invest more time in preparation, prevention, planning, 
and capacity-building, we reduce emergency time in the first quadrant. The survey found that companies with high 
performance and efficiency tend to spend 65-80% of their energy on the second quadrant and 15% on the third. 
Poorly run companies tend to do the opposite. 


B. Support Technology 
The main technologies involved in this study are shown in Table 2. 
TABLE 2... DEVELOPMENT TECHNOLOGY LIST 


Develop Content Technical Support 


Develepient Microsoft Visual Studio 2005 
Environment 
Data Storage Microsoft SQL Server 2005 


Interface Processing Adobe Photoshop, Html. DIV. CSS 
Function Development | Asp.Net, C#, JavaScript, Ajax 


Development Data access layer, business logic layer, interface 
Architecture presentation layer 
System Framework B/S mode 


III. ANALYSIS AND DESIGN OF TIME MANAGEMENT SYSTEM FOR COLLEGE STUDENTS 


A. Common software analysis 

Through the analysis of the existing time management software such as “Memorandum”, “Calendar” and “Plan 
Manual”, we found that most of them have the following problems: they are mainly based on the first three 
generations time management concept, their functional orientation is not consistent with the situation of college 
students, and they have complex operations or unclear interfaces. Therefore, it is necessary to design a time 
management system which has a high level of management performance, meets the actual needs of college students 
and has a good operation experience. 
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B. Demand analysis 
1) User demand analysis 
A questionnaire survey on the time utilization and demand of college students was conducted, and the 
questionnaire effective recovery rate reached 92%. The content and results of the survey includes five aspects: 

@ Personal information of the respondents. Among the respondents, girls make up 86% and boys make up 
14%. Undergraduates accounted for 79% and postgraduates accounted for 21%. The samples are distributed in all 
grades and various professions, and have certain representativeness. 

@ The basic situation of time use. More than 60% of the students spend most of their time on 
“learning”, ”living” and “entertaining”, “learning” accounted for the largest proportion of them. College students 
usually have more discretionary time than primary and middle school students, but nearly half of them think they 
feel busy during that time. 

@ The customary way of time management. Among the respondents, 60% have a clear goal, but only 1/3 
have detailed plans, only 1/3 can carry out the plan at a high level, and less than 30% are satisfied with the time 
management plan. This can be summarized as “more goals, less planning, less execution and poor satisfaction”. 
Fortunately, more than half of the students have the intention to improve their time management ability by learning. 

@ Compliance with the fourth generation of time management concept. 97% of the students are accustomed 
to taking urgency rather than importance of affairs as the preferred dimension, which does not conform to the fourth 
generation time management concept. 

@ The application of time management software. 88% of the students have never used time management 
software, and more than half are willing to try. Thus, the time management software for college students has a great 
potential for application. 


2) Functional requirements analysis 
The system is put forward to solve the problems of students' time and personal events, curriculum 
arrangement and so on. The key lies in the analysis of student time, especially the time distribution of college 
students. The existing software has not in-depth analysis of this, and this research as the main content. Based on the 
statistical analysis of the questionnaire and the fourth generation time management concept, specific functions of 
this system include: 

@ Set up a course in combination with the learning situation. The most frequently used type of affairs for 
college students is learning affairs. Class is an essential and important matter for students, and it is also a part of the 
system to be carefully designed. The system should provide the setting of the schedule, the inquiry of the schedule, 
and be able to distinguish the information of single and double weeks, and provide the information of teachers and 
classrooms in class. 

@ Set and handle the information of life affairs. College students! life and other affairs occupy a lot of time, 
and there are differences in the roles they play. It will play an important role in the time management of college 
students to handle daily affairs well. Therefore, for daily non-learning transactions, you should be able to provide 
Settings, and can query and manage. Transaction setting is a large part of daily time management. Transaction 
setting should have starting and ending time. When setting, it should be able to distinguish its importance from its 
urgency. 

@ Set roles and schedule tasks based on goals. According to the fourth generation of time management 
theory, transactions should be set according to individual roles. The system should provide life goals, term goals, etc, 
and determine transactions based on goals and roles. 

@ Set priorities and reminders for important affairs. The system needs to be able to clearly distinguish the 
important transactions in the learning and life affairs, mark and remind the important transactions, and promote the 
user to execute the plan around the important transactions. 

@ Provide the ability to query scheduling. The system should provide weekly scheduling queries, daily time 
utilization, event sequencing, etc. Weeks time can intuitive tell users this week schedule use of information and the 
time utilization intuitive way to tell users, today how many time is to use on the useful things, how much time can 
also be used. 

@ Supervise records of implementation results and follow up on training habits. According to the fourth 
generation of time management standards, transactions should have timely feedback summary. The problem of 
college students' bad habit of wasting time is more prominent, because it is difficult to change the bad habit formed 
gradually after the relaxation of the pursuit of study. Therefore, the system should be able to timely monitor and 
record the implementation results, and track the cultivation of habits. 

@ Provide software usage assistance and supports time management. Questionnaire statistical results show 
that the majority of students and have not used the time management software, so the application of the software 


364 


should be provided tutorial, its content is set in line with the principle of concise, the user can quickly master the 
software usage, and establish the use of interest. 


C. System design guidelines and basic principles 

The main task of college students is to master knowledge and grow up healthily. Considering that their time is 
flexible and variable, the system design should give full consideration to the specific situation of college students' 
learning, choosing, making friends and healthy life, as well as make a balanced allocation of them. Specific 
principles include: 

@ Harmony and consistency. The core aim of time management is to improve college students' effective 
comprehensive coordinated development, finally realizes the personal knowledge level, physical quality, 
psychological quality, mental outlook, the harmonious development of the communication ability and so on. In the 
process of system design, different time arrangements of college students in study, life, family and other aspects 
should be taken into account to provide various management approaches for college students. 

@ Balance functions. In view of the different types of time management, there should be a relatively unified 
form of classification, so that users can easily look at the overall situation, to balance the use of each part of the 
situation, and do not overemphasize the neglect of a certain type of business. In the design of the system, the steps of 
goal, event and feedback should be separated into separate blocks, so that the system is in accordance with the order 
of thinking in form. 

@ Surrounding center. The system design should take individual differences into account, and provide each 
college student with personalized central tasks, so that all their goals and plans are around their central tasks. In the 
system, students should be explicitly provided with life goals and term goals, and help analyze their roles and 
responsibilities. 

@ Flexible. The design of the system should reserve certain modification space. For time-sensitive and 
time-sensitive transactions, it should be able to adjust timely and minimize the impact on other plans. Provide 
recording and feedback capabilities for event execution. 


D. The function module design 
According to the analysis of functional requirements, the system mainly includes five major modules, as 
showed in Figure 1. 


College Student Time Management System 


Query and Reminder Executive Feedback Help Support 
Module Module Module 


Target Module Event Module 


‘ 5 * A * A * time " 

connection target learning . time event timed implementation habit learning use modify 
aa look at role life events | | schedule ae Fi E iv : : 2 management 

mission events events utilization sorting reminder correction formation diary tutorials a Passwords 


Fig. 1. The system structure 


The modules are as follows: 

@ Target Module. Function of this module is: Users input the life ideal, semester goal, role and other content, 
from the macro direction to grasp the university time. Provide the event name for the formulation of "target event", 
provide the basis for "daily event" and "learning event", and do not involve the time and information of specific 
event. 

@ Event Module. Function of this module is: The user enters information such as start, end time and repeat 
period of a specific event here to determine the specific time distribution of the event. 

@® Query and Reminder Module. Function of this module is: Visually query the five-day time schedule, 
provide the user with an intuitive calendar schedule, and provide a time utilization ratio. 

@ = Executive Feedback Module. Function of this module is: Help the user to record events, habits, to the user 
feedback for the user to record time use of the space result of experience, objectives, plans to provide the reference 
for the next step for the user. 

@ Help Support Module. Function of this module is: Use all functions of the system for users and provide a 
time management tutorial for users. Users can view this part of content at any time during use. 

The target module is a prerequisite for scheduling time. Users should first determine their life goals, 
semester goals, etc., and set their personal roles and role matters according to the mission information. Then, the 
target event is determined according to the role transaction guidance, and the learning event and daily event are 
made according to the personal role. The event information generates the time schedule and executes the plan 
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according to the schedule. Finally, the implementation of time arrangement is recorded, and the habit cultivation is 


obtained by systematic statistics. Help the support module support the overall time utilization. The relationship 
between each module is shown in Figure 2. 


| generate Time Execution 
Target Event =|~>——> Help Support 
/ XS Arrangement / Feedback \ ibis 
pe T / \ 
x / \ 
\ / \ 
% L 
Contact the | guide | Lookatthe | guide Execution 


fetes Target Event Habit Record statistical 
mission role 


record; Outcome | statistical 


Fig. 2. The relationship between each module 


E. Workflow design 

After determining the functional modules of the system, the process design of the time management system 
for college students is carried out in accordance with the process sequence when it is used. The process is shown in 
Figure 3. 

If you don't have an account, you need to register before you log on to the system. After logging into the 
system, if there is any modification to the target or event, it will enter the corresponding page operation. You can 
query the schedule in the system and execute it according to the schedule. After the execution of the event, the 
execution status and experience of the event can be filled in. The system will record and count the execution status. 
You can exit the system at the end of the access. 
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Fig. 3. The workflow 
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F. Database design 
The database used by the time management system for college students is SQL server2005. The database 
name is "db_TimeManage", and the data tables contained in the database are shown in Table 3. 


TABLE 3. _ Database: db_TimeManage 


Order | Table Name Functional Description 
1 tb_User User Table 
2 tb_Role Role Table 
3 tb_Course Course Table 
4 tb_Task Event Table 
5 tb_Execute Executive Condition Table 
6 tb_ Diary User Diary Table 
7 tb Resource System tutorial and Time tutorial 


The data table contents of database db_TimeManage are as follows: 

@ tb User. Used to record the user information, including login user name, password, and other related fields 
and life ideal, mission statement, universities, semester goals a, second and third term target term target a few fields. 
Where the user number is the primary key, it is also the identity column. 

@ tb Role. It is used to record the user's role and role goals, including fields such as user sequence number, 
role, target 1, target 2 and target 3. A user can set up up to seven role records. Where the role number is the identity 
column. 

@ tb Course. Used for storing course information, including user serial number, course serial number, course 
start and end time, class start and end time, class name, teacher, classroom, single and double week, etc. The course 
number is the main key. 

@ tb task. Used to store events, including user sequence number, event start and end time, name, remark, 
repetition, habit formation, importance, emergency, etc., where event sequence number is the primary key. 

@ tb Execute. Used to record event execution, including event date, event sequence number, execution, etc., 
where the execution sequence number is the primary key. 

@ tb Diary. Used to record user diary, including user serial number, diary date time, diary title, diary content, 
etc., in which the diary serial number is the main key. 

@ tb Resource. Store system tutorials and time tutorials, including resource Numbers, resource titles, 
resource types, resource content, and other fields, where resource Numbers are the primary key. 


G. Interface presentation layer design 
On the basis of functional module design, the page effect diagram after login system is designed. The 
interface strives to be clear, compact, harmonious and has a good interaction experience. The design effect is shown 


in figure 4. 
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Fig. 4. Page design renderings 
On the main page, there are site logo bar, navigation bar, notification area, work area, footer information and 
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other areas. Among them, the function selection is realized by clicking the link and the drop-down menu on the 
navigation bar. The work area USES the iframe framework. When selecting the function, the information of other 
areas is retained. The content of the operation is displayed in this section, which is the main area of the page. 


IV. IMPLEMENTATION OF TIME MANAGEMENT SYSTEM FOR COLLEGE STUDENTS 
This chapter is on the basis of system design, using the related system development technology, to realize each 


function module of system, mainly including the login module, object module, the event module, query module, 
feedback module, help support module page. Main pages of the system are shown in Figure. 5, 6, 7 respectively. 
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Fig. 5. The system directory 
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Fig.7. The system's front page 
V. SUMMARY AND PROSPECT 
A. Summary 
Based on the fourth generation of time management theory, this paper studies the current situation and 
characteristics of college students' time use, and analyzes the existing problems of existing time management 
software. On this basis, summarizes the time using the actual demand of college students, put forward the design 
scheme of the system, and use ASPNET, ADO.NET, JavaScript and other technology to achieve the college 
students’ time management based on the theory of the fourth generation time management system. The system is 
simple in operation and suitable for college students to arrange and plan their time in a scientific and reasonable way. 
The system can not only provide prior time setting, adjust the time plan in time, but also carry out executive 
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feedback and follow up afterwards. 


B. Rpospect 

The preliminary application shows that the system can have certain positive effects on promoting the 
formation of good behavior habits of college students and establishing scientific concept of time management. The 
research can also be expanded from the following aspects: 

Firstly, the influence of social, family, personal preference and other factors are also included in the 
formulation and implementation of the plan. The following research can continue to go deep in the aspects of 
college students undefined behavior trend, personality characteristics, learning style and so on, and gain more user 
application feedback. 

Secondly, the convenience of operation is an important factor affecting the user experience. The 
development of the system can introduce more graphics drag-and-drop operations, create a more harmonious 
interface, make the software more easily accepted by college students, and further improve the efficiency of 
management time. 

Thirdly, this system focuses on the design and implementation of time management function. With the 
popularization of intelligent mobile devices in college students, the next research can be combined with the 
development of mobile platform. 
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Abstract 


Blended learning, drawing from best practices in both online and face-to-face learning, is on the rise in higher 
education. Research shows that faculty needs professional development and support to get prepared for successfully 
developing and teaching blended courses. This article will share how an institution has re-developed a training 
program on blended learning based on learning theories and faculty feedback, along with its innovation for an 
effective process and greater learning impact. 


Introduction 


Blended learning, drawing from best practices in both online and face-to-face learning, is on the rise at 
colleges and universities. Campus Technology conducted its first “Teaching with Technology” survey in 2016, and 
71% of faculty respondents reported using a mix of online and face-to-face environments to teach. The New Media 
Consortium Horizon Report (2017) has identified blended learning design as a top trend to drive technology 
adoption in higher education. 

Northern Virginia Community College is the largest public educational institution in Virginia and the 
second-largest community college in the United States. Because of the heavy traffic issue due to its location in the 
nation’s capital metro area with six campuses, the institution has an increasing need for blended courses. To meet 
this special need and prepare faculty to teach blended courses, NOVA Online has re-developed a training program 
on blended learning offered for all the college faculty. The mission of NOVA Online is to design and implement 
excellent and innovative instruction and student support services for online learners. It also supports instructional 
technology initiatives college-wide through certification programs, courses, workshops, small group consultation, 
and individualized training. 

The re-developed training program is a three-week online course offered at Blackboard as Blackboard has 
been the learning management system that is used in the college. Since the online course needs extensive knowledge 
and skills in Blackboard, Blackboard proficiency is required as a prerequisite for the workshop on blended teaching 
and learning. 


Literature Review 


Blended learning is the “thoughtful integration of classroom face-to-face learning experiences with online 
learning experiences” (Garrison & Kanuka, 2004). Blended learning describes “learning activities that involve a 
systematic combination of co-present (face-to-face) interactions and technologically mediated interactions between 
students, teachers and learning resources” (Bliuc, Goodyear, & Ellis, 2007, p. 234). Blended learning also denotes a 
reduction in face-to-face contact or seat time (Vaugham, 2007; Picciano, 2009; Mayadass & Picciano, 2007). For 
example, Picciano (2009) included “‘a portion” (institutionally defined) of face-to-face time [be] replaced by online 
activity” (p. 10). Allen and Seaman (2016) categorized traditional teaching as having 0% of content delivered online 
and blended teaching as having 30-79% of content delivered online, but the definition appears to be broad and 
vague. However, researchers recognized benefits of a broadly structured definition of blended learning since it 
allows institutions to adapt and use the term as they see fit and develop ownership of it (Sharpe, Benefield, Roberts, 
& Francis, 2006, p. 17). Dziuban, Hartman, and Moskal (2004) argued that “blended learning should be viewed as a 
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pedagogical approach that combines the effectiveness and socialization opportunities of the classroom with the 
technologically enhanced active learning possibilities of the online environment, rather than a ratio of delivery 
modalities” (p. 3). Heckman, Osterlund, and Saltz (2015) pointed out blended learning frameworks are often 
anchored by the dimensions of time and place, describing interaction modes as either same time or different time 
and either same place or different place. They stated that “both such a focus on delivery mode could sometimes run 
the risk of being instructor-centered or content-centered, rather than student-centered” (p. 4). 

Blended learning has many benefits from the perspectives of instructors, students, and institutions. Graham, 
Allen, and Ure (2005) described three general benefits of blended learning: 1) enhanced pedagogy and learning 
effectiveness, 2) increased access and flexibility, and 3) improved cost-effectiveness and resource use. Blended 
courses make instructors explore new and different ways to teach by integrating activities from both face-to-face and 
online learning environments. Blended learning may provide pedagogical benefits such as increased learning 
effectiveness, satisfaction and efficacy (Garrison & Kanuka, 2004; Graham, 2013; Means, Toyama, Murphy, Bakia, 
& Jones, 2009). Blended learning provides students increased access to higher education offerings because of its 
convenience, less seat time, and a flexible schedule (Vaughan, 2007). Blended learning also save institutional 
resources such as classroom space and parking space, and thus make teaching and learning more cost-effective 
(Graham, 2013; Moskal, Dziuban, & Hartman, 2013). 

Lloyd-Smith (2010) claimed that blended learning has emerged as a potential solution to address the 
diverse learning needs of community college students. Community college students tend to have multiple 
responsibilities outside of school, making flexible education an important determinant. Blended courses offer the 
convenience and flexibility of fully online courses without the loss of faculty and student personal interaction (Sitter 
et al., 2009). Xu and Jaggars (2011) found that students from blended courses were similar to those from face-to- 
face classes and they equally tended to complete their courses. Blended learning provides students social 
connections that enhance communication, thereby supporting student retention and success (Hijazi, Crowley, Smith, 
& Shaffer, 2006). Studies indicate that blended courses can be effective in promoting student success, but only if 
they are designed and delivered with care. 

For faculty who have never taught online, blended courses can be challenging to design and teach as they 
need to develop new technological and pedagogical skills for this teaching modality. Research shows that 
professional development is crucial to prepare and support faculty to teach blended courses (Owens, 2012). Whether 
they take a low-impact blend or a medium-impact blend or a high-impact blend design approach (Alammary, 
Sheard, & Carbone, 2014), faculty must have the technological skills to design and maintain the online portions of 
blended courses. They also must have pedagogical skills needed for instructional methods unique to blended 
learning. Dziuban, Hartman, and Moskal (2004) concluded that maximizing success in blended learning requires a 
planned and well-supported approach that includes a high-quality faculty development. 


Re-development and Impact 


Constructivists believe that learning is an active process in which learners construct new ideas or concepts 
based upon their current and prior knowledge. Vygostky’s zone of proximal development indicates that learning is a 
social process in which people can develop more skills through peer interaction or collaboration than learning alone. 
Since adult learners are self-directed learners and instruction for adult learners should be task-oriented and related to 
the real world, adult learning programs should be developed to capitalize on the experience of all participants. 

Constructivism and adult learning theories guided the re-development of the training program from its face- 
to-face format into a three-week online format offered on Blackboard. The initial training program required faculty 
to attend a series of face-to-face workshops on blended learning and then develop a blended course for review to 
earn a certificate of completion. The training program would often take faculty more than one semester to complete. 
The training process was long and the workload was very high for both faculty participants and professional training 
staff in terms of teaching and learning, course development, course review, and training data record management. 
Feedback from faculty and professional staff showed an ineffective training program. To enhance the training 
program and increase the learning impact, the training program has been redeveloped into a three-week online 
course that integrates active learning, social learning, project-based learning, and authentic learning into a new 
program to better meet the institutional need. 

In addition to active learning supported by constructivism and social learning supported by Vygostky’s 
zone of proximal development, the online course also addresses authentic learning and project-based learning to 
make up for the initial course development and course review components in the previous face-to-face training 
program. Authentic learning is often referred to as real-life learning that is associated with a real-world problem or 
situation. It encourages learners to create a tangible and useful product, and provides learners with opportunities to 
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connect directly with the real world beyond the classroom. Project-based learning is an instructional approach that is 
centered on learners. It is defined as “an instructional approach that contextualizes learning by presenting learners 
with problems to solve or products to develop” (Moss & Van Duzer, 1998, p. 1). Project-based learning is different 
from traditional teaching as it emphasizes learning through learner-centered, interdisciplinary, and integrated 
activities in real world situations (Blumenfeld et al., 1991; Solomon, 2003). 

Besides constructivism, project-based learning is also theoretically anchored in situated cognition (Brown, 
Collins, and Duguid, 1989). Brown, Collins, and Duguid (1989) think that learning is maximized if the context for 
learning resembles the real-life context in which learning materials will be used. Learning is minimized if the 
context for learning is not similar to the context in which the learning will be used. Project-based learning integrates 
experiences, knowledge, real-world contexts and situations into an active learning process for the creation of a 
tangible artifact, which produces meaningful learning. 

The three-week online course covers all the basics about blended learning that the initial face-to-face 
workshops offered, and it also provides faculty an opportunity for active learning, social learning, project-based 
learning, and authentic learning without an additional course review. Three weeks of online learning makes faculty 
commit a certain amount of time to finish all the learning activities and assignments during the three weeks. Two out 
of three weeks provide faculty with opportunities to discuss and share topics such as understanding of blended 
learning, benefits and challenges of blended learning, examples of blended learning activities, and issues faced while 
designing a blended course. Taking a project-based learning and authentic learning approach, the online course 
requires faculty to develop and create artifacts such as a blended learning syllabus, a blended learning course design 
blueprint, and three modules of blended learning for a real-world course faculty have taught in a face-to-face format 
or a new course faculty are going to teach as a blended course. Furthermore, all the learning materials are accessible 
to all learners with captions available for all the videos in the course. After the three weeks of online learning, 
faculty will gain basics about blended learning as well as course design and hybrid pedagogy through hands-on 
learning at Blackboard. 

After faculty complete the online course, they will receive a certificate of completion through the online 
registration system. At the end of the online course, anonymous survey is conducted with all the faculty participants 
to collect feedback about the online course and their learning experience. Below is a brief summary of course 
evaluation surveys conducted with 304 faculty participants from Spring 2016 through Fall 2017: 


Survey Items Mean (n=304) 
Requirements for successfully completing the workshop were clearly stated. 4.7 
The workload was appropriate. 4.5 
I was able to regularly share ideas with others. 4.4 
I feel prepared to develop a blended course now. 4.6 
The workshop increased my confidence in my ability to teach a hybrid course. 4.5 
I would like to recommend this workshop to a friend or colleague. 5.0 
I’m satisfied with the workshop. 4.7 


The survey is composed of questions with a Likert scale from 5 points for “Strongly Agree” to 1 point for 
“Strongly Disagree” and open-ended questions for faculty to share comments and thoughts about the online course. 
During the two years from Spring 2016 through Fall 2017, 304 faculty have completed the online course and 184 
faculty have completed the survey, which yields a response rate of 61%. The survey data shows a very positive 
result about clear requirements of the workshop communicated to faculty, appropriate workload, peer learning 
through regular sharing with other fellow faculty members, feeling prepared to develop a blended course, and 
increased confidence to teach a blended course. All faculty respondents were very satisfied with the workshop and 
all of them would like to recommend the workshop to a friend or a colleague. Some qualitative data from the open- 
ended questions in the survey are listed below to show faculty’s positive experience about the online format of 
training, timely response from the facilitator, peer learning opportunity, project-based learning, and the online 
course itself. 


Samples of positive comments on the online training format: 
What I liked most was the simply ability to participate at any point and at any hour. That makes a huge 
difference to me in being able to complete the workshop. I think the course is fine as is, likewise the 
technology. 
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I really like the online format. That’s the only way it can make me complete the hybrid training course 
while teaching my own students. 


Samples of positive comments on the facilitator of the online course: 


I appreciated the instructor's readiness in dealing with questions the class had and the regular 
communication. 

I was and am very satisfied with the course. Professor Wang was and is a great Instructor. She is very 
warm, lovely, and friendly. She works very well with everybody. She is patient, understanding, and 
understands human problems. She is highly educated, very competent, and very good at what she does. A 
great Instructor, and a great asset to NOVA. 

I really liked the quick, detailed feedback from the instructor. I enjoyed the discussion boards and 
collaboration with fellow faculty. I cannot think of anything specific to improve the course. 

Excellent course. The optional materials are extremely helpful, and applicable. Anyone not well versed in 
hybrid pedagogy would benefit from the course. The instructor is extremely helpful and provides very 
timely and appropriate feedback. I especially appreciate the quick responses to emails (same day, usually 
within a short period of time). 


Samples of positive comments on peer learning opportunities: 


What I like most was the ability to share with others and get a wider view of the difference between hybrid 
and online courses. 

I like the fact that this workshop used discussion boards and required its students to reply to another 
students thread, I think that is an excellent idea. I like the outlines explaining what is going to happen in 
class and online. 

With regards to what I liked most, I thought the implementation of the discussion boards was particularly 
nice because it allowed me to get a sense of how other disciplines are implementing the hybrid format into 
their classrooms. 


Samples of positive feedback on the course: 


When I see that a course is well organized and easy to navigate, I know that it's going to be a good 

course. Then when I reviewed the content for each module, I knew that it would help me as I design the 
online component of my courses. 

I liked everything about the course. The videos and readings were interesting and to the point. Hong 
communicated with us often and was very approachable. The technology was not over my head which was 
a major relief. I also appreciated that Hong said to start out using what we know and add to it later down 
the line. 

I love the way the instructor gives us detailed instructions and it made it easier for us to understand what 
was expected from us for each assignment. I am very happy to learn about the resources available to us via 
this course. Thank you. 

I liked the assignments that help to frame the shape of an actual course that we would create. 

The course was informative and Hong's instructions were very clear and easy to follow. 


In addition to re-developing the face-to-face workshops into an online course, the training registration 
system has also been improved to make the registration process and data management easier. The initial registration 
was through emails and then an online system. Training staff had to spend lots of time to trouble shoot technical 
issues for registration and manage the training data in spreadsheets. After the online system is changed into a home- 
grown system for registration and data record management, more time is saved from technical support for 
registration and more effective data record management is achieved. 
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Conclusion 


Innovation takes many forms in both theory and practice, and its core value lies in problem solving and 
evolving (Magna & Buhan, 2018). Examples of this type of innovation for evolving include researching new and 
better ways to enhance teaching, changing education delivery to appeal to a new target population, and 
implementing new ideas to bring about better outcomes. Sustaining innovation refers to a process, system or 
modification that improves an existing product or system. It makes the process better and more efficient as well as 
more beneficial to the end users. Our sustaining innovation is achieved through both course re-development and new 
registration system development. By redeveloping the face-to-face workshops into a three-week online course with a 
focus on authentic learning, peer learning and project-based learning, data showed increased learning access, 
learning effectiveness, and faculty satisfaction and thus better support them to teach blended courses. By developing 
a new registration system to manage enrollment roster, course evaluation survey and certificates of completion, 
operation process showed reduced time in technical support for course registration, training data management, and 
certificates management. Our intention to improve training and solve a problem faced in the training program 
resulted in a greater impact in both faculty learning and training process management. 
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Two descriptors: blended learning, community of practice 
Abstract 


Blended learning continues to be on the rise in higher education. Research shows that faculty needs professional 
development and support to successfully develop and teach blended courses. This article will share how a 
community of practice supported faculty for effective practices in blended courses and what lessons learned for 
future implementation. All colleagues passionate for student success in blended learning can learn and enlighten 
their own practice. 


Introduction 


Blended learning, drawing from best practices in both online and face-to-face learning, is on the rise at 
colleges and universities. Campus Technology conducted its first “Teaching with Technology” survey in 2016, and 
71% of faculty respondents reported using a mix of online and face-to-face environments to teach. The New Media 
Consortium Horizon Report (2017) has identified blended learning design as a top trend to drive technology 
adoption in higher education. 

Northern Virginia Community College (NOVA) is the largest public educational institution in Virginia and 
the second-largest community college in the United States. Because of the busy traffic issue due to its location in the 
nation’s capital metro area with six campuses, the institution has an increasing need for blended courses. While the 
college offers a three-week online course multiple times per semester to prepare faculty for teaching blended 
courses, there are no other continuing programs to support faculty when they are teaching blended courses in the real 
world. After each session of the online course is offered in the college, feedback is collected from faculty 
participants regarding their learning experience and suggestions for future improvement of this training program. 

Although the feedback is generally very positive, a number of faculty indicated that some continued 
support after the online course is necessary if possible. The authors of this article, guided by the blending with 
purpose model (Picciano, 2009), developed a series of face-to-face seminars to continue support for faculty on one 
of NOVA’s six campuses in collaboration with NOVA Online that offers the online course college wide and 
Woodbridge Campus on which the onsite seminars were offered. This collaborative initiative in Spring 2018 
intended to promote effective practices in teaching blended courses through a community of practice. 


Literature Review 


Blended learning is the “thoughtful integration of classroom face-to-face learning experiences with online 
learning experiences” (Garrison & Kanuka, 2004). The Online Learning Consortium defines blended courses as 
courses that integrate online with traditional face-to-face class activities in a planned, pedagogically valuable 
manner, where a portion of face-to-face time is replaced by online activity. Allen and Seaman (2016) categorized 
traditional teaching as having 0% of content delivered online and blended teaching as having 30-79% of content 
delivered online, but the definition appears to be broad and vague. Researchers recognized benefits of a broadly 
structured definition of blended learning since it allows institutions to adapt and use the term as they see fit and 
develop ownership of it (Sharpe, et al., 2006, p. 17). Dziuban, Hartman, and Moskal (2004) argued that “blended 
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learning should be viewed as a pedagogical approach that combines the effectiveness and socialization opportunities 
of the classroom with the technologically enhanced active learning possibilities of the online environment, rather 
than a ratio of delivery modalities” (p. 3). 

Blended learning provides students increased access to higher education because of its convenience, less 
seat time, and flexible schedule. Researches have also shown that blended learning can enhance student engagement 
and learning outcomes (Dziuban, Hartman, Cavanagh, & Moskal, 2011) as well as students’ satisfactions (Martinez- 
Caro & Campuzano-Bolarin, 2011). For faculty who have never taught online, blended courses can be challenging 
to design and teach as they need to develop new technological and pedagogical skills for this teaching modality. 
Research shows that professional development is crucial to prepare and support faculty to teach blended courses 
(Owens, 2012). Faculty must have the technological skills to design and maintain the online portions of blended 
courses. They also must have pedagogical skills needed for instructional methods unique to blended learning (Korr, 
Derwin, Greene, & Sokoloff, 2012). 

Picciano (2009) identifies a blending with purpose model composed of six components: content, social 
/emotional, dialectic/questioning, collaboration/student generated content, reflection, and synthesis/ evaluation. 
Content is the primary driver of teaching and there are many ways in which content can be delivered and presented 
in a blended course. The model also indicates that teaching is not always just about learning content or a skill but is 
also about supporting students socially and emotionally. Dialectic /questioning is an important activity allowing 
faculty to probe what students know and help them construct their own knowledge. Collaborative learning, 
incorporating reflection, can be a powerful pedagogical strategy. The last element is synthesizing, evaluating and 
assessing learning. Although not every course needs to require students for group work and not every course should 
rely entirely on reflective activities, Picciano (2009) stated that the six components of the model should blend 
together in an integrated manner that appears as seamless as possible for students in a blended course. The learning 
objectives of a course should drive the activities and approaches of the model. 

Picciano’s blending with purpose model guided the design and development of four conversational 
seminars on effective practices in blended courses, with a focus on an overview of blended learning and the 
seminars, developing and blending content, developing and blending interaction and collaboration, and developing 
and blending assignments and assessment for each session respectively. The sessions are open to all faculty on 
Woodbridge Campus, with a preference to faculty who have completed the college’s online training program and 
who have already taught blended courses in the real world. 


A Community of Practice 


The initial concept of community of practice originated from “Situated learning: Legitimate peripheral 
participation” (Wenger & Lave, 1991). Wenger and Lave think learning is “increasing participation in communities 
of practice” (Lave & Wenger, 1991, p. 49). In his groundbreaking book, “Communities of practice: Learning, 
meaning, and identity,” Wenger (1998) expanded the idea of community of practice, articulating how social 
resources shape people’s learning trajectories and their professional identity. Wenger thinks a community of practice 
is formed by people who engage in a process of collective learning in a shared domain of human endeavor. It refers 
to a group of people who share a concern or a passion for something they do and learn how to do it better as they 
interact regularly with each other. The practice of a community is dynamic and involves learning from everyone. 


Henri and Pudelko (2008) classified four levels of communities considering the strength of a group’s social 
bonds and the extent of its intentionality: communities of interest, goal oriented communities, a learner’s 
community, and communities of practice. Communities of interest generate knowledge solely for individual use. 
Goal-oriented communities, driven by external forces, carry out particular tasks within a specific timeframe. A 
learner’s community relies on the instructor for guidance and it generates both individual and shared products. 
Communities of practice are organized around professionals. The community members perform similar activities 
and use strong social bonds and high levels of intentionality to extend and improve their practices by generating and 
building a shared knowledge. 

Wenger, Trayner, and de Laat (2011) presented the concept of value creation as a way to describe and 
assess social learning in a community of practice. The value is created as a result of community members’ activities 
and interactions with others. They defined five different cycles of value creation generated within a community of 
practice: immediate value, potential value, applied value, realized value, and reframed value. Immediate value refers 
to learning that is put to use immediately to solve a problem. Potential value refers to benefits related to shared 
knowledge and skills that can be realized at some time in the future. Applied value refers to application of shared 
knowledge and skills to new contexts. Realized value refers to knowledge and skills gained through a community of 
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practice made a difference in their ability to achieve important goals. Reframed value refers to the identification and 
definition of new criteria for success. 


Data Collection and Analysis 


The seminars were offered during the spring semester in 2018. Unfortunately, one of the buildings had 
flood emergency at the beginning of the semester. The campus was closed for one week, and almost 800 courses 
were moved and re-scheduled for the rest of the semester. While everyone was dealing with emergency, the seminar 
attendance rate was dropped. With all the efforts put into the initiative, three seminars were offered to cover all the 


planned content. 


A total of ten faculty attended the three different seminars. Since each time had different faculty 
participants, we interviewed four faculty members who have participated in all the sessions. They are all female 
faculty, but they teach four different subjects and all of them have teaching experiences in blended courses. The 
interviews were scheduled right after the end of the spring semester in 2018. Each interview lasted for one hour in a 
quiet study room and a set of same questions were given to each of the four faculty members to answer. Further 
questions and comments occurred during the interview conversation. Four interviews were recorded with Speechy 
App for later transcription. It took about four hours to transcribe each recorded interview. 

A brief summary of interview data is presented in the following table: 


she felt about 
her blended 
classes. 


new emerging 
technology 
available for 


Themes Faculty A Faculty B Faculty C Faculty D 
Faculty background | e Sheteachesart | @ She teaches e She teaches e She teaches 
history courses. English courses math courses. chemistry courses. 
e She has been as an adjunct e She has been e She has been 
teaching faculty. teaching teaching blended 
blended courses | @ She has been blended courses courses since 2004. 
since the teaching since 2012. e She is the first 
college started blended courses faculty to start 
offering. since 2011. teaching chemistry 
blended courses. 
Insights about e@ She thinks most | e She likes e Shethinks that | e She thinks 
blended learning students like blended blended engagement is a lot 
blended learning and learning is better with the 
courses. thinks it’s great absolutely required online or 
e “Obviously less for marine necessary but out-of-classroom 
time in the students. it’s not for learning 
classroom is e She is everyone. component. 
attractive concerned e Blended e There is a little bit 
because they about not courses more preparation in 
would rather be giving enough increase more blended courses 
there for less instruction in online compared with 
time.” blended courses components for face-to-face 
and feels students courses. 
insecure when working on 
not in the themselves. 
classroom 
engaging with 
students. 
Reasons for e Helpful to e Drawn to the e To find some e To learn something 
attending seminars know what seminars new ideas for new, in particular, 
other professors because of the teaching. about technology 
are doing. insecurities that | e To know what tools. 
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have 


in the discipline 


e To improve her implementing 
class for the into her 
students to teaching. 
make their 
English 
experience 
better. 
What they have EdPuzzle e Technology New concepts Some really useful 
learned Backward tools — the ways New materials from the 
design she can make it technologies presentations 
approach more Conversations Learning different 
interactive with with colleagues perspectives from 
students. to discuss people in different 
e The online challenges and disciplines 
lecture strategies Changed former 
component perspective about 
e Backward science teaching 
design 
approach 
What not working Just a little too e Wishing more She wanted to A little bit more 
for the seminars heavy on theory is done with the know specific hands-on probably 
tools math hybrid needed for some 
e Trying to tailor courses. sessions. 
things working 
for each 
disciplines is 
very difficult. 
Whether seminars Positive e Learned a lot It works Absolutely. It’s 
have met e Networking through actually way 
expectations e A lot of tools introducing beyond that. 
e Theory part is new concepts 
helpful. and new 
technology as 
well as 
discussing the 
challenges 
What else do you The same kind | @ More Published new She is eager to set 
expect to learn or of interchange technology ideas and up more similar 
suggestions you of ideas tools specific courses sessions more 


people can get 
involved in because 
it’s going to help 
everyone grow 


Discussion and Conclusion 


Our observation of faculty’s engagement in the conversation during the seminars and the interview data 
showed a very positive experience for faculty participants in general. All faculty have extensive experience in 
teaching blended courses, but they still think peer learning through conversational seminars is helpful. All of them 
want to learn what other professors are doing and what new ideas and new technology tools available to enhance their 
teaching. One faculty also shared her feeling of insecurity about teaching blended courses and expected to learn more 
through the seminars. The lessons we learned include 1) importance of community building; 2) incorporating both 
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design and technology components into the professional development program; and 3) balancing theory and practice 
with an emphasis on providing concrete examples from different disciplines. 

All faculty interviewees liked the conversational seminars and the peer learning opportunity through 
networking with others and community building. Their learning include new pedagogical ideas such as different 
methods about assessment, new technology tools such as EdPuzzle, discussions on challenges and issues along with 
social networking with other faculty. 

When asked whether the seminars have met their expectations, 

Faculty D said: 

“Absolutely. It’s actually way beyond that. I did not anticipate in learning much when I first came in. I didn’t 

anticipate but I did get more out of it.” 

Faculty B said: 

“T definitely learned a lot. One of my goals was definitely to network and that was 100% succeed. I 

definitely gained a lot of tools. The seminar was trying to tell us to do more interaction, and what it really 

shined was when you were showing us how to do this like the EdPuzzle tool.” 

Faculty C indicated importance of peer learning and community building: 

“T’m not sure why my other colleagues didn’t show on the seminars. We don’t have enough conversation 
with each other. Everyone is doing on our own. Nobody knows what others are doing. That’s the missing 
part.” 

When developing the seminars, we tried to incorporate both design and technology into the sessions. 

Faculty’s feedback showed the need of both components with a special favor on backward design approach 

and EdPuzzle as a technology tool. 

Faculty A remarked: 

“Hong mentioned that working backwards, like thinking about what you want them to know, develop 

assessment, and then build your lectures around that. I would do it in the other way so yeah that was just like 

something that stuck in my head ever since, that just makes sense and I never thought about it before.” 

Faculty B commented: 

“During the seminar, we discussed that thinking backwards, looking at the outcomes first, and then go 

backwards to identify content and design the activities, that’s very helpful.” 

Faculty B also shared her wish for sharing more technology tools and Faculty D would prefer some hands-on 
work for some sessions. 

Faculty C summarized her learning: 

“Overview part is absolutely mandatory to help think about what components are important for hybrid, and 
then technology. The most important conversation with colleagues.” 

While all faculty interviewees think theory is necessary, they shared a need to learn from practice through 
publication and concrete examples. 

Faculty B remarked: 

“In the beginning, theory part is helpful. It was interesting and it did make me think about things in a 

different way.” 

Faculty A indicated a little too heavy on theory while Faculty C shared her wish to learn more through 
specific examples: 

“T especially want to know specific math hybrid courses. Not just how people teach math hybrid in general, 
I would like to know something published and presented some new ideas.” 

Interview data showed that it is important to make a proper balance between theory and practice with a 

focus on providing concrete examples from different disciplines. 


In summary, the social bond among the faculty participants appeared to be strong and the extent of 
intentionality of the conversational seminars was high, which resulted in a community of practice (Henri & Pudelko, 
2003). Although the interview data indicated that none of the faculty participants have tried the new ideas and new 
tools in their teaching right after the spring semester, the data showed that they are going to try the new knowledge 
and skills in the future semesters and the seminars also made a difference in their ability to teach blended courses. The 
seminars did not seem to have created immediate value and applied value, but they appeared to have created potential 
value and realized value through the community of practice (Dziuban, Hartman, Cavanagh, & Moskal, 2011). This 
collaborative initiative to promote effective practices in blended courses through a community of practice proves to be 
helpful and effective although faculty participation is not very high. Faculty feedback is also encouraging and 
constructive. The lessons we learned from our initial effort will lead to better planned sessions for our faculty through 
a community of practice in the future. 
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Abstract 


This study investigated the experiences of international students in a doctoral instructional design and development 
program at a public university in the United States. Instructional practices differ globally, which present challenges 
as students become accustomed the new concepts and applications of their host country (Zhu and Flaitz, 2005). The 
study addresses the effectiveness of blended online coursework for international students, the ability to acquire 
professional competencies, the perceptions, and realities of experiences in the academic program. Findings suggest 
that degree of social and academic challenges for international students in the IDD program are reflected by 
motivations for enrolling, specifically autonomous and non-autonomous motivations 


International Graduate Student Perspectives & Experiences within an Instructional Design & 
Development Program: An Exploratory Examination 


The purpose of this phenomenological study was to reveal the experiences and changes in international 
students’ values and beliefs as they engaged in a doctoral instructional design and development (IDD) program at a 
public university in the southeastern United States (U.S.). The research included participants from diverse 
backgrounds and cultures who have different motivations for selecting instructional design (ID) as their field of 
study. 

The increasing number of books and academic articles published about the experiences of international 
students, as well as the appearance of the Journal for International Students dedicated to research on international 
students, is evidence of the growing academic interest in this population (Hackett, 2014). The data for this study 
were collected from international students currently enrolled in the program, as well as program alumni, to gain a 
deeper understanding of the needs and expectations of international students and will allow for the investigation of 
how to meet those specific needs and expectations. 

Regardless of the country of origin, international students in higher educational institutions in the U.S. share 
similarities in the challenges they face (Geary, 2016). For example, the delivery of instruction can be quite different 
than what they were accustomed to in their home countries, as are the academic demands and expectations for 
students. A lack of awareness of support services, if such services are available, can decrease motivation or overall 
satisfaction in a program of study for this population (Perry, Lausch, Weatherford, Goeken, & Almendares, 2017). 
The language barrier tends to be one of the biggest obstacles leading to adjustment issues. The nuances of American 
English lead to miscommunication and misunderstandings, affecting not only friendships with peers but also 
academic success (Leong, 2015). American students sometimes misinterpret behaviors of international students 
which may lead to isolation and embarrassment, especially in the context of online learning. 

There are issues associated with international students’ learning experiences that were not as prevalent until the 
emergence of web-based courses. Technological advances have made it more accessible for individuals of diverse 
populations to attend graduate courses in the U.S. For example, web-based instructional content may be a major 
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convenience for international students. However, self-expression of the understanding of course content in an online 
format could cause complications for individuals when English is the second language (Lui, Lui, & Magjuka, 2010). 

A new international student’s level of perceived competence come from feelings of frustration or feelings 
of effectiveness of one’s efficacy. Feedback of success or failure influence the new international student’s sense of 
competence. Language skill and cultural adaptation are anxiety-provoking factors for many international students 
which can decrease the international students’ sense of competence. 

To help build a sense of competence for someone from another culture, there are several approaches that 
can be taken. Teachers can make a concentrated effort to monitor the international students’ progress, strengths, and 
weaknesses and provide timely, informative feedback about their performances. International students should be 
encouraged to voluntarily participate in class and give rationales about their educational experiences, expressing 
their ideas freely. Also, collaborating with other students enhances the learning experience for students from other 
cultures as well as local students. Collaborative activities provide opportunities for international students to express 
their ideas, receive feedback from peers, develop global competencies for local students, and allows the international 
student to develop relationships necessary for cross-cultural understanding. The actions support effective-relevant 
feedback, allow students to grasp the meaning and worth or external goals, and provide the understanding and 
relevant skills conducive developing a sense of motivation. 

The IDD program provides a trajectory for scholarship through online research, forming relationships with 
academic advisors and peers in the field, and on-the-job learning through graduate assistantships. As graduate 
assistants, many these students work in teaching and learning centers which provide instructional design assistance 
to faculty. Such assistantships allow for a deeper engagement with the content of their coursework through hands-on 
experience. The IDD program provides opportunities to gain knowledge of the academic culture of their host 
country. Yet some international students lack experience with instructional practices that, in some cases, differ 
greatly from what they accustomed to in their home countries. There exists a potential to impact ID practices in their 
home country from the experiences with technologies related to learning and teaching that the IDD programs 
provide. Also, for those who plan to stay in the United States, the diverse backgrounds and academic experiences 
that international students bring with them can lead to unique contributions to ID research and practice in this 
country. 

The existing research on international students enrolled in American universities is vast, however, the global 
diversity of students in ID programs is mostly overlooked. When IDD students complete a doctoral program, a set of 
standard competencies are expected to be met (Koszalka, Russ-Eft, & Reiser, 2013), but does a gap exists in regard 
to international students? Our study addresses this gap and provides international perspectives on research in 
educational technology. 

This phenomenological study examined the perceptions of students from different countries in a blended ID 
program regarding their academic experiences. Three questions were addressed in the study: 

e What are the perspectives of international students related to ID research? 

e How effective is the blended nature of the ID program with this special population? 

e How do the expectations for the program prior to the international student’s arrival to the U.S. compare 

to the realities faced once immersed in the IDD program? 


Method 
Participants 


The participants for this study were graduate students in an IDD Ph.D. program at a university in the 
Southeastern United States and alumni. There were seven participants. Five participants were female and two were 
male. There were two alumni and five current IDD Ph.D. students. The participants were from South American and 
Asian countries. 

Procedure 


The appropriate approach for such exploratory research was a comparative type of collective case study. We 
collected data from seven current students and graduates from different cultures to reveal the similarities and 
differences of their experiences. The complexities of the integration of the study of ID and the international 
experience were revealed with the use of qualitative interviewing, observations, and journal reflection. Each case 
was examined in total, and then the different cases were compared in a cross-case analysis for patterns that cut 
across the cases as well as differences. 
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Results 


There were five themes in the qualitative analyses: Enhanced professional development and assistantships. 
Enhanced pre-arrival orientation materials, motivations for why the students enrolled in the IDD program, the 
relationships with advisors, mentors, and peers, and academic flexibility. 


Quotes from Participants 
Learning Contexts 


“Tt was difficult at first. I was not sure what was expected of me. I didn’t know the appropriate 
amount to write on the forums and since the classes were online, I didn’t want to offend any of my 
classmates. Communicating online, I was afraid to use the wrong words or my tone might come off as 
aggressive. Also, my instructors assumed that I knew how to use the LMS and was accustomed to learning 
in an online environment. Even though I was an educator in my home country, we did not have access to 
the internet or most other technologies that are taken for granted. I wish the university would have 
provided some guidance or training for new international students so that we would not have to spend the 
first few months catching up and feeling lost. All new students would benefit from these training, 
especially international students.” 


-IDD program alumnus, female, China, working as an instructional designer in the United States 
Pre-arrival expectations, post-arrival realities 


“Before I arrived in the United States, I expected that education was so much better than my home country, 
simply because it was... American, which is assumed to be superior. I found the education setting to be 
much different than what I was used to back home, which was no surprise, but I expected much more 
guidance in America. There is a lot of flexibility and independence, which is great for me. However, I 
could see there being problems for those who cannot adapt to the openness of American education. For 
instance, I took it up myself to find trainings and workshops at the university’s teaching and learning 
center. The workshops were intended for faculty, but knew I needed to understand the LMS so I could be a 
more efficient student. I found the trainings on my own, and asked permission to attend as a student. If I 
were not persistent, I would not have had that opportunity. Mentors are of the utmost importance for 
success, and I’m not just talking about your advisor; just someone who cares about your academic growth 
and a person whom you trust. To be successful international students need these relationships but not all 
know where to even start.” 


- Current IDD student, female, Honduras, working as a graduate assistant 
Learning Content and Learning Contexts: 


“The online learning experience was easy. A little too easy, actually. However, I hold an MS for a 
university in the States. Even though my master’s program was an entirely face-to-face program and the 
IDD program blended, my previous learning experiences prepared me for what to expect. I believe the 
online learning experience in this program is ineffective because it requires so little effort and the 
instructors are hardly present. I have learned a lot about instructional design but mostly from working with 
my advisor, as well as what I learned through my graduate assistantship”. 


-Current IDD student, male, India, working as a graduate assistant 
ID Research 


“There is a heavy focus on theory, which is very important, but the program could use more opportunities 
for student research in instructional design. I am the only instructional designer at this company. They 
highly recognize my expertise and are aware of the needs of ID knowledge and skills. Of the knowledge 
and skills that I acquired in this program, I use adult learning, needs assessment processes, qualitative 
analysis, and evaluation.” 


-IDD program alumnus, male, Thailand, currently an instructional designer in his home country 
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The findings aligned with Deci and Ryan’s (2017) Self-Determination Theory (SDT), the theory defines 
autonomy as the need to self-regulate one’s experiences and actions. This self-regulation satisfies the need to feel 
that an individual is in control of their own behavior. Autonomous behaviors are self-endorsed with an individual’s 
authentic interests and values. The basic psychological need of autonomy is associated with volitional, congruent, 
and integrated feelings. This means that the need for autonomy is satisfied when an individual makes decisions of 
their own will and takes actions that are consistent with one’s ideal or actual self. 

Ryan and Deci (2017) are careful to distinguish autonomy from the closely-related concept of self-reliance. 
While both autonomy and self-reliance influence an individual’s decisions and behavior within a social context, 
autonomy is one’s ability to make choices that follow an individual’s personal will, and self-reliance is negotiating 
between the point where external forces and personal drive intersect. 

Self-reliance, or independence, differs from autonomy in SDT because only some intentional actions are 
truly autonomous. Self-reliance is the degree to which actions are influenced by external forces such as other people, 
tules, laws, and social environments. Autonomy is the sense that behaviors are self-endorsed, ego-centric acts, 
reflecting one’s will. While self-reliance is the degree to which actions are not fully endorsed by the whole self and 
does not totally represent the preferences and values of the individual. Self-reliant behaviors are influenced by 
external influences which allow an individual to trust in one’s capabilities, judgement, or resources. Additionally, 
autonomous behavior can be thwarted if the individual feels pressured or coerced by forces perceived to be outside 
one’s self (DeHaan, Hirai, & Ryan, 2015). 

For many psychologists, the concept of autonomy differs from what is defined in SDT. Cross-cultural 
psychologists define autonomy as independence or individualism. The SDT concept of autonomy relies on the 
volition and willingness of an individual as motivation for behavior; it does not refer to independence as autonomy 
(Chen et al., 2014). With this in mind, different cultural contexts can provide diverse world views by which the need 
of autonomy is influenced. For example, students from collectivist-oriented societies may feel autonomous when 
following the advice from others for whom they have respect. Students from individualistic-oriented cultures will 
feel autonomous by making their own decisions and expressing personal experiences. 


It was found that the degree of social and academic challenges for international students in the IDD 
program are reflected by motivations for enrolling, specifically autonomous and non-autonomous motivations (see 
Figure 1.) 
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Figure 1. Motivations and Resulting Experiences of Enrollment 


Discussion 


To attract and maintain a talented and academically successful international student presence on campus, 
variability of the psychological needs across cultures must be explored. Professional development opportunities 
should be available to provide an understanding of underlying assumptions of SDT. These opportunities could be 
targeted for faculty and staff who work closely with international students so that they can be prepared to help build 
a sense of autonomy for international students as they adapt to their new academic environment. Development of 
trainings which deal with culture-specific perspectives of autonomy (e.g. the Hutsulian-Ukrainian concept of 
autonomy) may be too resource consuming. However, assumptions can be made about the social-cultural contexts 
across different cultures that can help international student support professionals facilitate internalization and satisfy 
the need for autonomy. 
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Involvement and the quality of interpersonal relationships with teachers, advisors, and peers will influence 
the need for affiliation and belongingness for international students in a new educational setting. Another influential 
factor is the cultural distance between the student’s native culture and the dominant culture within the learning 
environment. Students from countries that are culturally similar to the United States may more quickly and easily 
feel more connected to their peers, academic practices, and course content in comparison to students from culturally 
distant countries. Also, pre-arrival expectations and post-arrival realities may also conflict, which could affect the 
sense of relatedness. 

Teachers and advisors can help build a sense of relatedness only if they try to understand and acknowledge 
international students’ feelings and perspectives. Building relationships that offer constructive feedback, but accept 
expression of negative reactions, allows the international student to feel that they are valued in the academic 
program. Empathizing with unpleasant feelings in a way that provides comfort can increase the quality of 
interpersonal relationships between international students and faculty. 

Professional development and work opportunities (e.g. graduate assistantships) can increase a sense of 
relatedness by creating environments that facilitates affiliation, belongingness, and connection with others. This 
approach can integrate international students into the university system and establish formal or informal peer-to-peer 
mentoring. Such relationships can provide a constructive and positive connection through friendly and professional 
involvement, heightening the sense of affiliation and relatedness while mitigating relational exclusion and 
loneliness. 

There were limitations to this study. More international students and alumni should be interviewed for a 
deeper examination of the perspectives, expectations, and the effectiveness of the IDD program. Faculty should also 
be interviewed and given a chance to reflect on what they are doing to improve relationships and learning 
experiences of international students, and data should be collected from multiple IDD programs to see if findings are 
similar across different universities and programs. 
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The Collaborative Instructional Design System (CIDS) is an innovative tool in instructional design that will 
benefit teachers as well as stakeholders for both schools and higher institutions. 


The Collaborative Instructional Design System deals on the important aspect of the 21*- century instructional 
design with wider perspectives involving various communities in education, sharing and collaborating ideas and 
strategies, promotes creativity while establishing “globally competitive learners” towards the era of IR4.0 for a better 
future living in a new emergence of smart Society 5.0 


Abstract 


The transformation of the education ecosystem is critically required particularly in the era of the 
information technology. The main aim is to enhance the quality of educational practices whilst creating an inspiring 
learning environment for learners, giving them an opportunity to determine their own learning activities as far as 
learner-centered approach is in practice. Much effort has been spent in providing e-learning applications to the 
learners, tools, and strategies for teachers. However, not much effort has been done in connecting the dots - 
integrating elements and requirements of the current educational needs, fulfilling the nation’s educational policies 
and aspirations into a mechanism that helps teachers to have a wider perspective in the process of designing the 
instructions creatively, systematically, practically and professionally. 

The Collaborative Instructional Design System (CIDS) is an innovative tool in instructional design that will 
benefit teachers as well as stakeholders for both schools and higher institutions. It was developed as an option for 
educationalist in fulfilling the current educational needs especially the 21st Century education with its 4Cs learning 
needs (critical thinker, communicator, collaborator, creator), and learning opportunities developing “globally 
competitive learners”, regardless of their abilities. It is a new dimension at engaging teachers and other professional 
learning communities to be collaboratively involved in the 21st-century learning, facilitating and preparing learners 
to the Fourth Industrial Revolution — IR4.0, while facing the challenges of the newly emerging smart Society 5.0. 

CIDS comprises The Integral ASIE Instructional Design Model (Ismail Md. Zain, Balakrishnan M. 2014, 
2016). It’s IHE features (integrative, hybrid, eclectic) has the capacity to collaborate with many elements related to 
education in producing a highly-rich, effective and creative instructional planning activities. Aspects of the model 
are based on the proposed 21*-Century Learning Framework (21* Century Partnership, 2002) and Four-Dimensional 
Education (Fadel C, Bialik. M., Trilling B. (2015). It has fundamentally encompassed the philosophical attributes of 
metaphysics, epistemology, axiology, ethics, and logic. These philosophical underpinnings strengthen the need for 
professional education player primarily classroom teachers to execute this model in their daily teaching and learning 
endeavors. It is classroom-based ID model that follow the theories of behaviorism, cognitivism, constructivism 
(Jonassen, 1991) and connectivism (Siemens, 2005; Ireland, 2007) and advanced features of the Professional 
Learning Community - - PLC, (Richard D.F. (2004). It enhances teachers’ professionalism while enriching learners’ 
experiences connecting virtually with other communities. It is accessible at https://asiemodel.net 


Introduction to the collaborative instructional design system - Connecting the Dots 


The Collaborative Instructional Design System (CIDS) is a system that gives teachers an option at 
designing the current teaching and learning environment in connecting the dots - integrating elements and 
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requirements of the current educational needs, addressing the IR4.0 and Society 5.0 issues and fulfilling the nation’s 
educational policies and aspirations into a mechanism that helps teachers to have a wider perspective in the process 
of designing the instructions creatively, systematically, practically and professionally. It comprises The Integral 
ASIE Instructional Design Model (Exhibit 1), a transformation of instructional design model 


(http://edtechreview.in/trends-insights/insights/ 1058-instructional-design-models-in-the-2 | st-century-a-review), that 
provides a simple and practical planning tool fulfilling the features of current and future education needs. It allows 
teachers strategize approaches, methods, and activities for learners to determine their own choices that will inspire 
them at engaging with learning activities creatively, joyfully while preserving the characteristics of teachers' 
professionalism in learning and facilitating procedures. It is a constructive process in designing the instructions, 
which provides practitioners in education a valuable tool and perspective in enhancing the quality of instructions for 
all learners regardless of their capabilities as well in fulfilling the 21st-century learning requirements. 

This model provides the procedural flow of the instructional planning which is flexible, constructive and 
user-friendly. It provides an option for practitioners in the educational field as a valuable tool or mechanism in 
planning the lesson creatively, following the needs of learners based on their characteristics or attributes. Learners 
are also given an equal opportunity to determine a variety of methods, activities, and recommendations proposed to 
be experienced in an enjoyable and exciting learning and facilitating environments. This model has a broad 
perspective towards improving the quality of learning, facilitating and training activities. It provides opportunities 
for teachers to discuss and share materials, experiences and their creativity with other colleagues throughout the 
country towards creating a high-tech learning approach fulfilling the requirements of the current and future 
education landscape. While special advanced features of the Professional Learning Community (PLC) (Exhibit 2) 
with its wider scope and concepts are integrated into CIDS giving more opportunities for teachers, learners, and 
communities to be connected and share valuable information on various aspects of education especially related to 
the employability issues. Individuals are invited or voluntarily participate in the program by registering as a member 
of PLC. They can choose to become members of a particular community group - educators, administrators, teachers, 
professional, student, and private communities. Users may communicate virtually with PLC members for advice, 
contributions, and sharing of ideas in meeting the needs of learning skills. 


Exhibit 1: The Integral ASIE Instructional Design Model 
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Theoretical and conceptual framework of the collaborative instructional design system 


The Integral ASIE ID Model has fundamentally encompassed the philosophical attributes of metaphysics, 
epistemology, axiology, ethics, and logic. This can be seen in flexible planning items that rely heavily on the 
creativity of teachers and students. These philosophical underpinnings strengthen the need for professional 
education player primarily classroom teachers to execute this model in their daily teaching and learning endeavors. 
It can be seen from various perspectives, concepts, and theories in the following descriptions. Theoretically, by 
looking at various perspectives, CIDS, as its name applied, has the capacity to collaborate with many elements in 
the instructional planning. The Integral ASIE Instructional Design Model encompassed in CIDS has the features of 
IHE (integrative, hybrid, eclectic). The items contained in this model are integral in nature because they are 
integrated with the various basic elements of education, which include the science of technology, pedagogy, and 


Exhibit 2: The Main Professional Learning Community Page 
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content knowledge - TPACK (Mishra, P., & Koehler, MJ (2006) comprehensively to meet the current learning 
needs. Looking at the features on the components and items of the model, it can also be classified into the hybrid 
instructional design system category though basically it is a classroom-based ID model but it has a wider scope 
which goes far beyond the four walls involving the virtual environments learning which give rooms to the blended 
type of learning as well as to make the possibility of flipped classroom being practiced. It follows an eclectic 
approach to the instructional design whereby a designer (user) blends ideas from multiple learning theories to 
construct a learning experience that works better than from only one theoretical influence. Hence, the paradigms of 
behaviorism, cognitivism, constructivism, and connectivism are likely being considered and applied in the various 
instructional planning procedures, unlike some of the conventional models that initially designed for Instructional 
System Development (ISD) (Seel, N. M. 1997, Gustafson K.L., Branch R.M. 2002) which make this model differs 
from other conventional ID models. 

Theories of behaviorism, cognitivism, constructivism are the three broad learning theories most often 
utilized in the creation of instructional environments. These theories, however, were developed in a time when 
technology did not impact learning. New technology forces the 21‘-century learner to process and apply 
information in a very different way and at a very different pace from any other time in history thus, lead to the 
emergence of connectivism. According to Siemens, (2005) connectivism was driven by the understanding that 
decisions were based on rapidly altering foundations. New information is continually acquired, and the ability to 
draw distinctions between important and unimportant information is vital. As a result of the above theories, various 
learning and teaching models were developed which introduce different methods and techniques to be applied by 
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teachers and students in learning situations. These methods and techniques that can be integrated into teacher's 
planning procedures according to their own creativity in the Integral ASTE ID Model. While adapting to the basic 
principles of instructional design and other instructional design models such as Dick & Reiser Model, Dick & Carey 
Model, ASSURE Model, ARCS Model, ADDIE Model, attention was also given to other related fields in the 
formation of this ID model. Basically, what distinguishes between The Integral ASIE ID Model with conventional 
ID models is in terms of the goal towards meeting the concept of the learner-centered approach. Most other models 
carry the question "What elements need to be determined by the teacher to build a learning situation for learners" - 
is more teacher-centered because teachers determine the planning process. While for The Integral ASIE ID Model 
focus "How teachers strategize the learning in developing a flexible learning situation to meet to the needs of the 
learners" - it is more learner-centered learning because teachers only suggest various elements that are appropriate 
while learners are given the opportunity to implement those activities that inspire their interest in establishing “fun- 
learning” environment. 

A special attention has been given to learners' profiles, their readiness in following the lesson, media 
attributes, and learners thinking levels in establishing the “differentiated learning” situation. Moreover, the relevance 
between the principles and theories taught in the classroom with the working environment to be met in the era of the 
Revolutionary Industry 4.0 (IR4.0) and their position and role in the emergence of the smart society 5.0 (Society 
5.0) in the future, is a vital factor to be analyzed, strategized and implemented. These factors are important in 
creating the concept of “thinking out of the box” among learners and educationalist as well, in addressing to the 
current education developments. Likewise, the key elements in the current learning framework such as the 21st 
Century Learning Framework, (2002), Four-Dimensional Learning, (2015) and Framework for 21st Century 
Learning, Ministry of Education, Malaysia (2017), that have a close correlation between each other are also part of 
the planning items in this model to ensure its relevant to current learning situations. Practically based on the 
theoretical description, teachers require skill in pedagogical knowledge, technological knowledge, and content 
knowledge — TPACK (Mishra, P., & Koehler, MJ 2006) integrated into their learning and facilitating activities. A 
Strategic Learning Structure Framework (Ismail Md Zain, 2016), (Exhibit 3) was developed to clarify how the 
components of instructional planning in The Integral ASIE ID Model were formed and displayed in a Multiple 
Integration Worksheet (MIW) based on the areas in the TPACK Model (Exhibit 4). 


Exhibit 3: A Strategic Learning Structure Framework 
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The 21%century education 


So much attention has been given on the characteristics and features of the 21*' Century Learning 
framework in the development of the Integral ASTE ID Model (Exhibit 5) because it is kind of current requirements 
in learning that support the era of IR4.0 as well as the emergence of the smart Society 5.0. While in 2015, The Four- 
Dimensional Education (Fadel C., Bialik. M., Trilling B. (2015) has been released which focused on the 
transformation of education systems in relation to the competencies and aspirations needed by our learners in the 
21*-century learning environment. Being closely related to the 21‘ Century Partnership framework, the component 
has been restructured to portray the qualities of human learning dimensions namely the knowledge, skills, and 
character that involved the metacognition processes. 


Exhibit 4: Components of TPACK Model in The Multiple Integration Worksheet 
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The Integral ASIE Instructional Design Model — the procedural application 


The Integral ASIE Instructional Design Model is an integrated strategical approach to the designing of the 
instruction in fulfilling the learners’ needs in today's learning environment. ASIE refers to the components of 
Analyze, Strategize, Implement, and Evaluate (Exhibit 1). Each component has several aspects and items 
accordingly (Exhibit 6). This “learner-centered approach” interactive online ID model in designing instruction, 
engaged learners of different abilities exploring and unleashing their potentials in generating and creating ideas 
through Higher Order Thinking Skills (HOTS) activities, (Ismail Md Zain, 2013). The model is interactive - a user- 
friendly interactive features which provide options to interact with the content of the items, integrative in planning 
the content - provide options for teachers to integrate planning items provided or add new items as needed in the 
planning process, prescriptive - provide a fast accessing of preset information that assists users in understanding the 
features of the model, and constructive in the organization of the components - a complete, comprehensive and well 
organized strategical planning procedures. The model begins with the individual teacher or a group of teachers in the 
same subject area collaboratively analyzed the suggested aspects in the first component of the model that include the 
instructional profiles such as subject, theme, learning areas, topic, learning objectives and outcomes or other profiles 
for the particular subject. In the second component of the model, features of the 21st-century learning skills 
analyzed, various instructional tools (techniques, methods, and activities) selected, appropriate thinking tools 
especially dealing with the principle of higher order thinking skills (Ismail Md. Zain, M. Balakrishnan, 2014), and 
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aspect of moral values identified. This leads to the formation of instructional questions. They are essential questions 
for the topic which formed instructional strategies in the instructional planning. Teachers will select the strategies 
prepared in the third component for their best practices in the classroom to ensure its appropriateness and 
effectiveness for the learners. The final component is the evaluation stage whereby responses from feedback are 
gathered to review and revise the instructional planning strategies in the respective component and aspect of the 
model. It is a reflection process for future instructional redesigning opportunities. However, evaluation is not only 
taking place at the end of the planning but at every component of the model as indicated by dotted lines. The 
Reflection Cycle is another form of evaluation/reflection for teachers in their instructional planning process as 
indicated by the green circle. There are other advanced features that benefit teachers in their instructional planning 
as well as instructional leaders (principle, headmasters, evaluators) in monitoring, supervising, evaluating and 
accessing the teachers’ professional competencies. There are options allowing teachers to create daily lesson plans 
(DLPs) according to their respective needs creatively and professionally and creating a Professional Learning 
Community (PLC) to build character and unleash learners' creativity. 


Exhibit 5: The relationship of the 21‘*century learning features integrated into the components of The Integral ASIE 
ID Model 
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Multiple Integration Worksheet (MIW) 


An Integral ASIE ID Model has its own displayed planning worksheet known as Multiple Integration 
Worksheet (MIW) (Exhibit 4 & Exhibit 7). It gathers a wide instructional planning information by users through the 
customized aspects and items in the components of the model. These unique features guide teachers in the process of 
formulating & integrating the best possible practices for learners with different abilities in the instructional planning 
strategies at the macro level (overall planning for a specific topic) and at micro levels (creation of daily lesson plan — 
DLP). Since teachers are encouraged to plan their instruction collaboratively with their colleges, thus, MIW helps 
teachers in the lesson study session (features of Professional Learning Community — PLC) to determine the best 
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possible item that fit well in their classroom practices. It is editable and savable in the pdf file format to the user while 
providing options in requesting it to be shareable among others across the nation. Information selected or written in 
the 1$' & 2"! component of the model (Analyze & Strategize) is compiled or inserted in the MIW. It will be displayed 
when the user clicks on any aspects of the 3™ and 4" component of the model (Implement & Evaluate). 


Exhibit 6: Components, Aspects, and Items of The Integral ASIE ID Model 
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Exhibit 7: Multiple Integration Worksheet (MIW) 
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Daily Lesson Plan (DLP) — Planning at Micro Level 


The content of DLP is in 2 parts following the model’s component (Exhibit 8). The first part is the information 
of the selected items from the MIW- 1‘ & 2" component of the model (Analyze & Strategize). The second part is the 
information of the teaching and learning activities (facilitating activities, learners’ engagement activities) as indicated 


in the 3r4 component of the model (Implement) as well as impact, reflection, and remark as indicated in the 4" 
component of the model (Evaluate). 


Summary - The Impact of CIDS on teachers, learners and administrators - Strength and Capabilities 


Generally, it is time effective, reduce teachers’ burden in preparing the lesson, provide sharing of ideas, 


establishing unlearn, learned and relearn society - to learn different approaches in educational designing, relearn of 
new 
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NAME: NABILAH BINTI JAMIL 
CLASS LEVEL: YEAR 5 
CLASS: 5 ARIF 

SUBJECT. ENGLISH 

WEEK: 18 

DATE: 30-04-2018 

TIME: 11:00 AM TO 12:00 PM 


Exhibit 8: Daily Lesson Plan (DLP) 
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Description 


strategies formulated in response to the changes in the educational landscape, and unlearn the past experiences through 
the transformation process in creating awareness of the importance and impact of current innovation in ID towards 
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lifelong learning as required by The Sustainable Development Goal 4 —Education 2030 Agenda — UNESCO, 
(UNESCO, 2014). Learners have equal opportunities for gaining knowledge & skills regardless of their differences 
in capabilities resulting from the high-quality systematic design of teachers’ instructions. Unleash their potentials 
through learners centered approached, established globally competitive learners — following the 21-century learning 
needs, Inspiring learners — learning are not just gaining knowledge and skills but responding to the industrial needs 
for future employability. School administrators are accessible for verification, monitoring, evaluating and identifying 
teachers’ competences (strength and weaknesses), assist in identifying the relevant CPD courses for teachers while 
establishing a complete network and database of teachers’ instructional planning. 
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